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Abstract 

The  allocation  of  resources  should  be  a  rational  decision  making  process  where 
alternatives  can  be  compared  based  on  their  estimated  costs  and  benefits  to  the 
organization.  In  order  to  justify  technology  transfer  activities,  a  sound  methodology  must 
be  developed  that  will  document  the  benefits  derived  from  transfer  activities.  The  risks 
or  imcertainties  associated  with  those  benefits  must  also  be  estimated  and  analyzed.  By 
detailing  the  costs,  benefits,  and  uncertainties  associated  with  technology  transfer 
activities,  decision  makers  will  have  a  logical  framework  that  can  be  used  to  determine 
the  cost  effectiveness  of  technology  transfer.  This  will  help  decision  makers  understand 
the  returns  on  investment  that  are  gained  by  the  DoD  through  technology  transfer 
activities. 

Leaders  within  the  technology  transfer  arena  are  searching  for  better  ways  of 
quantifying  the  tangible  and  intangible  benefits  of  technology  transfer.  The  goal  of  this 
research  is  to  build  an  acceptable  methodology  that  can  be  used  to  identify  and  quantify 
the  tangible  and  intangible  benefits  received  within  Air  Force  Materiel  Command 
(AFMC)  as  a  result  of  technology  transfers.  This  exploratory  study  employs  a  structured 
interview  methodology  to  identify  and  quantify  the  benefits  received  by  the  Air  Force 
through  its  technology  transfer  activities. 
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This  study  identified  several  findings  on  the  benefits  received  by  WL  from 
technology  transfer.  Most  notable:  The  benefits  received  through  technology  transfers  do 
match  up  with  the  benefits  expected  and  identified  by  the  AFMC  technology  transfer 
office;  a  majority  of  the  CRD  As  in  this  study  are  producing  revenues,  in  many  cases  the 
revenues  are  substantial;  and  clearer  objectives  are  needed  in  order  to  better  focus  future 
technology  transfer  activities.  In  addition,  this  researcher  provides  recommendations  to 
improve  the  measurement  of  benefits  received  through  technology  transfer  activities  and 
offers  future  research  opportunities  in  the  area  of  measuring  success  of  technology 
transfer. 
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AN  EXPLORATORY  STUDY  OF  THE  BENEFITS 


RECEIVED  BY  WRIGHT  LABORATORY  (WL)  FROM 
TECHNOLOGY  TRANSFER  ACTIVITIES 

T.  Introduction 

Background 

Recent  changes  to  the  Federal  Acquisition  Regulation  (FAR)  mandate  that 
Department  of  Defense  (DoD)  acquisition  strategies  must  account  for  and  focus  on  the 
commercial  applications  of  government-sponsored  technologies.  By  incorporating  such  a 
technology  transfer  process  into  its  acquisition  strategy,  the  DoD  will  be  able  to  better 
utilize  the  nation’s  research  and  development  (R&D)  resources  (Federal  Acquisition 
Regulation  1 996).  In  situations  when  a  developed  technology  may  or  may  not  be  used  to 
enhance  our  nation’s  warfighting  capabilities,  commercialization  of  those  government- 
sponsored  R&D  activities  is  a  means  of  realizing  a  greater  return  on  investment. 
Government-sponsored  R&D  can  often  be  used  by  a  private  firm  to  develop  new  products 
and  improve  its  competitiveness  within  a  given  industry.  This  situation  benefits  society 
as  a  whole  with  improved  socio-economic  benefits  to  the  nation  (16:  2). 

Technology  transfer  also  puts  an  additional  burden  on  military  professionals  who 
work  to  procure  weapons  systems  for  the  DoD.  Additional  resources  (manpower,  funds, 
and  materials)  are  used  in  the  technology  transfer  process.  In  an  era  of  downsizing, 
military  professionals  are  asked  to  do  more  with  less  and  senior  leaders  are  tasked  to 
allocate  resources  as  efficiently  and  effectively  as  possible.  Therefore,  committing 
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resources  to  technology  transfer  activities  takes  resources  away  from  other  research, 
development,  and  acquisition  activities.  This  requires  decision  makers  to  be  highly 
analytical  to  justify  the  allocation  of  resources  in  support  of  technology  transfer  efforts, 
since  those  resotirces  could  potentially  be  used  in  other  ways. 

The  allocation  of  resources  necessitates  a  rational  decision  making  process  where 
alternatives  can  be  compared  based  on  their  estimated  costs  and  benefits  to  the 
organization  and  the  nation  (38:  64).  In  order  to  justify  technology  transfer  activities,  a 
sound  methodology  should  be  utilized  that  will  document  the  benefits  derived  from 
transfer  activities.  The  risks  or  uncertainties  associated  with  those  benefits  should  also  be 
estimated  and  analyzed.  By  detailing  the  costs,  benefits,  and  uncertainties  associated 
with  technology  transfer  activities,  decision  makers  will  have  a  logical  framework  that 
can  be  used  to  determine  the  cost  effectiveness  of  technology  transfer.  The  goal  of  this 
research  is  to  develop  a  sound  methodology  that  can  be  utilized  by  senior  leaders  to 
accurately  calculate  the  benefits  of  technology  transfer.  This  will  help  decision  makers 
understand  the  returns  on  investment  that  are  gained  by  the  DoD  through  technology 

transfer  activities. 

Research  Objectives 

The  goal  of  technology  transfer  is  to  promote  the  use  of  technology  with  military 
applications  for  commercial  use.  Since  the  Federal  Technology  Transfer  Act  (FTTA)  of 
1 986,  technology  transfer  efforts  have  increased  dramatically  (31: 1 8;  5:  970).  The 
question  that  is  often  asked  is:  “What  do  government  agencies  receive  in  return  for  their 
efforts?”  This  is  a  difficult  question  to  answer  since  many  of  the  benefits  received  by  the 
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Department  of  Defense  and  the  federal  laboratories  are  difficult  to  quantify.  Because  the 
tangible  and  intangible  benefits  are  often  difficult  to  quantify,  the  rising  costs  attributed 
to  technology  transfer  efforts  are  becoming  more  difficult  to  justify. 

Much  research  has  discussed  and  documented  the  need  for  successful  technology 
transfer.  The  national  benefits  derived  from  technology  transfers  are  enormous,  but  in 
many  ways  unquantifiable  (16:  4).  Some  research  has  been  done  in  an  attempt  to 
quantify  the  national  benefits  derived  from  technology  transfers.  Unfortunately,  attempts 
to  directly  measure  the  national  benefits  derived  from  technology  transfers  and  quantify 
them  at  hierarchical  levels  has  been  unsuccessful  (16:  6).  Many  Air  Force  organizations 
have  collected  data  on  the  revenues  produced  by  transfers,  such  as  license  fees,  royalty 
payments,  or  leasing  fees  (16:  6-9).  Unfortunately,  there  have  been  no  efforts  to  link  the 
revenues  generated  by  transfers  to  the  specific  transfers  that  generated  the  revenue. 
Additionally,  there  have  been  few  efforts  to  quantify  the  intangible  benefits  received  by 
government  agencies  as  a  result  of  technology  transfers.  The  objectives  of  this  research 
are  as  follows: 

1.  Identify  and  document  methods  used  in  other  organizations  to  quantify  the 
tangible  and  intangible  benefits  received  from  technology  transfers. 

2.  Build  an  acceptable  methodology  that  can  be  used  to  identify  and  quantify  the 
tangible  and  intangible  benefits  received  within  Air  Force  Materiel  Command 
(AFMC)  as  a  result  of  technology  transfers. 

3.  Develop  a  methodology  that  will  estimate  the  expected  return  on  investment 
(ROI)  generated  by  technology  transfers  based  on  the  tangible  and  intangible 
benefits  derived  from  the  transfers.  This  methodology  must  take  into 
consideration  the  uncertainties  involved  with  estimating  the  benefits  derived 
from  a  given  transfer. 
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Scope 


This  research  focuses  primarily  on  the  development  of  processes  that  can  be 
employed  within  AFMC  to  justify  their  technology  transfer  efforts.  There  are  currently 
no  methodologies  in  place  to  accurately  measure  the  benefits  received  by  AFMC  as  a 
result  of  the  technology  transfer  program.  A  considerable  amount  of  resources  axe  used 
within  AFMC  to  transfer  the  technologies  developed  by  Air  Force  laboratories  to  the 
private  sector.  It  has  been  difficult  to  measure  the  return  on  the  resources  invested  in  the 
technology  transfer  program  because  current  measurement  techniques  are  inadequate  (16: 
14,15).  Benefits  received  are  difficult  to  identify  and  hard  to  quantify  once  they  have 
been  identified. 

Other  organizations  are  studied  to  identify  current  methodologies  being  used  to 
quantify  tangible  and  intangible  benefits.  Once  identified  and  validated,  these 
methodologies  are  applied  to  the  AFMC  technology  transfer  program  to  determine  their 
usefulness.  This  research  does  not  focus  on  identifying  the  inadequacies  of  the 
methodologies  used  by  other  organizations  in  quantifying  intangible  benefits.  The  sole 
purpose  is  to  identify  or  build  an  appropriate  methodology  that  can  be  employed  by  the 
AFMC  Technology  Transfer  Office  (TTO)  to  quantify  the  tangible  and  intangible 

benefits  of  their  technology  transfer  efforts. 

Definition  of  Terms 

The  following  section  lists  and  defines  several  key  concepts  that  are  used 
throughout  this  research.  The  definitions  listed  below  provide  the  reader  with  an 
overview  of  the  key  concepts  presented  in  this  paper  and  how  those  concepts  are  used. 
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Accounting  rate  of  return  -  A  measurement  of  a  project’s  contribution  to  the  net  income 
of  the  organization  (6:  391). 

Capital  budgeting  -  A  process  of  analyzing  potential  capital  expenditures  and  deciding 
which  investments  a  organization  should  undertake  (6:  384). 

Commercialization  -  Innovation  for  profit;  the  development  of  a  technology  into  a 
marketable  product,  service,  or  process  that  is  in  demand  in  the  private  sector. 

Developer  -  An  organization  (usually  a  federal  laboratory)  that  invents  or  develops  a 
technology  for  government  use. 

Hurdle  Rate  -  An  organization’s  cost  of  capital;  usually  the  interest  rate  charged  to  an 
organization  when  it  borrows  money  or  the  expected  return  on  an  alternative  project  (6: 
395). 

Internal  rate  of  return  -  A  discount  rate  which  results  in  the  present  value  of  a  project’s 
future  cash  inflows  (revenues)  to  equal  the  present  value  of  the  projects  cash  outflows 
(costs).  An  acceptable  project  would  have  an  IRR  greater  than  the  organization’s  cost  of 
capital  (6:  394). 

Net  present  value  -  The  value  of  a  project’s  cost  minus  the  sum  of  the  project’s  revenues 
after  the  revenues  are  discounted  at  the  project’s  cost  of  capital.  An  acceptable  project 
would  have  a  positive  net  present  value  (6:  392). 

Payback  period  -  The  expected  number  of  years  required  to  recover  the  original 
investment  in  a  project  (6:  388). 

Technology  sponsor  -  A  third  party  responsible  for  identifying  technologies  available  for 
transfer,  marketing,  and  actually  transferring  the  technology  (33:27). 

Market  pull  -  A  “needs”  generated  technology  transfer  process  model  where  there  is  a 
market  demand  for  a  specific  technology  developed  by  the  Air  Force  laboratories  (33:27). 

Technology  push  -  A  “means”  generated  technology  transfer  process  model  where  a 
specific  technology  is  developed  in  the  Air  Force  laboratories  and  supplied  to  the  market 
before  it  is  solicited  (33:  27). 

Technology  Transfer  -  The  dissemination  of  government-developed  technology  to  the 
private  sector  for  commercial  application. 
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Relevancy 


One  of  the  most  difficult  aspects  of  the  technology  transfer  process  is  that  the 
benefits  received  by  the  developer  are  not  obvious  to  the  casual  observer.  Prior  to 
making  any  capital  investment,  managers  in  the  business  world  analyze  the  potential 
returns  and  the  risks  associated  with  those  returns.  These  returns  are  then  measured 
against  the  potential  investment.  When  potential  returns  exceed  the  initial  investment, 
the  investment  is  justified.  Discounted  cash  flow  methodologies  like  net  present  value 
(NPV)  and  internal  rate  of  return  (IRR)  are  often  used  to  justify  a  capital  investment  (9: 
69;  25: 28).  Unfortunately,  the  technology  transfer  process  does  not  lend  itself  easily  to 
these  methodologies. 

Royalty  payments  or  licensing  fees  are  the  only  obvious  cash  revenues  received 
by  the  developer.  However,  these  payments  are  often  not  received  in  a  timely  manner.  In 
addition,  many  of  the  transfers  that  take  place  do  not  generate  any  revenues  in  the  form  of 
royalty  payments  or  licensing  fees  (16:  6-10).  These  transfers  benefit  the  developer  and 
society  as  a  whole,  but  the  benefits  are  often  intangible,  or  tangible  and  difficult  to 
quantify.  In  fact,  it  is  argued  that  the  main  reason  for  technology  transfer  is  the 
accumulation  of  socio-economic  benefits  and  not  cash  revenues  (16: 10).  The  goal  of 
technology  transfer  is  to  utilize  government-sponsored  R&D  technologies  as  effectively 
as  possible.  By  providing  technologies  to  the  private  sector,  the  government  provides 
private  firms  with  the  opportunity  to  commercialize  the  technology  and  provide 
additional  benefits  and  wealth  to  society  as  a  whole  (16:  6). 
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Leaders  within  the  technology  transfer  arena  are  searching  for  better  ways  of 
quantifying  the  tangible  and  intangible  benefits  of  technology  transfer.  By  identifying 
and  quantifying  the  benefits  or  non-cash  revenues  received  as  a  result  of  technology 
transfers,  future  technology  transfer  efforts  can  be  more  readily  justified. 

Thesis  Overview 

Chapter  Two  of  this  research  provides  backgroimd  information  on  the  technology 
transfer  process,  a  closer  look  at  the  policies  and  processes  that  have  been  implemented  to 
enhance  transfer  activities,  analysis  of  the  measurements  used  within  AFMC  to  measure 
technology  transfer  success,  a  discussion  on  current  return  on  investment  methodologies, 
and  a  look  at  existing  methodologies  used  in  the  private  sector  to  quantify  and  measure 
the  intangible  benefits  of  technology.  This  information  will  be  used  to  build  a 
methodology  for  identifying  and  quantifying  the  intangible  benefits  of  technology 
transfer  within  DoD. 

The  methodology  presented  in  Chapter  Three  will  be  an  exploratory  study  of 
technology  transfers  that  have  taken  place  within  Wright  Laboratory.  Interviews  will  be 
conducted  and  “encoded”  to  develop  estimates  of  the  tangible  and  intangible  benefits 
associated  with  transfer  activities.  These  benefits,  once  quantified,  will  be  combined  with 
the  cash  revenues  received  in  the  transfer  to  calculate  the  return  on  investment  (ROI) 
based  on  the  project’s  NPV.  Chapter  Four  will  present  the  data  collected  during  the 
interview  process  and  document  the  process  of  quantifying  the  data  and  using  it  in  the 
ROI  calculation.  Chapter  Five  will  present  the  conclusions  and  managerial  implications 
of  this  research  and  will  identify  an  agenda  for  future  research. 
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II.  Literature  Review 


Background 

The  overall  goal  of  the  technology  transfer  process  is  to  better  utilize  the  nation’s 
research  and  development  (R&D)  resources.  The  main  thrust  of  DoD-driven  R&D  is 
always  the  improvement  of  the  nation’s  war-fighting  capabilities.  However,  there  are 
instances  where  DoD-driven  technology  cannot  be  utilized  by  the  military.  In  these 
cases,  commercialization  of  government-sponsored  R&D  activities  is  the  best  means  of 
realizing  a  full  return  on  the  nation’s  R&D  dollars.  Government-sponsored  R&D  can 
often  be  used  by  private  firms  to  develop  new  products  and  improve  their 
competitiveness  within  a  given  industry  (38:  24;  33:  19;  26:  33).  This  situation  is 
beneficial  to  our  nation  as  a  whole  (16:  6). 

There  is  a  growing  perception  that  the  U.S.  is  not  getting  an  adequate  return  on  its 
federally-funded  R&D  budget  (33:  20).  This  may  be  an  accurate  assessment  for  two 
reasons.  It  is  certainly  possible  that  valuable  technologies  have  failed  to  reach  the  private 
sector,  depriving  society  of  benefits  and  the  government  of  licensing  fees  and  royalty 
payment  revenues.  However,  the  lack  of  appropriate  technology  transfer  effectiveness 
measures  is  a  more  likely  reason  for  the  afore  mentioned  perception  (33:  19-21;  16:  7- 
11).  Technology  transfer  organizations  are  being  tasked  to  develop  quantifiable  measures 
of  their  transfer  effectiveness  (33:  19-21).  In  an  effort  to  meet  this  challenge,  several 
different  methodologies  are  in  place  to  measure  the  effectiveness  across  a  variety  of 
technology  transfer  organizations.  Unfortunately,  this  scenario  has  led  to  a  lack  of 
measurement  standardization  among  different  organizations  (33:  19-21).  This  has  made 
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it  difficult  for  senior  decision  makers  to  assess  the  effectiveness  of  technology  transfer 
activities. 

As  a  result  of  DoD  “drawdowns,”  large  U.S.  federal  laboratories  are  faced  with 
cutbacks,  consolidation,  and  possible  closure.  This  environment  has  helped  to  increase 
the  importance  of  the  “technology  transfer  mission”  within  the  federal  laboratories.  The 
Clinton  administration’s  policy  towards  technology  transfer  openly  encourages 
collaboration  between  federal  laboratories  and  private  industry  (25:  64;  38:  26). 
Technology  transfer  is  not  a  new  concept;  however,  it  now  has  a  much  higher  profile  and 
is  more  important  than  ever  before  (25:  64).  This  high  profile  has  increased  the 
importance  of  more  effective,  quantifiable  measurements  of  technology  transfer  activities. 
The  purpose  of  this  research  will  be  to  develop  more  effective  measures  of  technology 
transfer  activities.  By  doing  this,  an  accurate  picture  of  the  effectiveness  of  the  Air 
Force’s  technology  transfer  program  can  be  developed. 

Technology  Transfer  Policies  and  Processes 

Congress  has  passed  a  series  of  legislation  aimed  at  improving  the 
commercialization  of  technologies  arising  fi*om  the  federal  investment  in  research  and 
development.  The  Stevenson- Wydler  Technology  Innovation  Act  of  1980  established 
technology  transfer  as  a  mission  ofthe  federal  government  (5:  970,971;31:  20;  22:  73; 

38: 23).  It  specifically  states  that  the  government  is  responsible  to  ensure  the  full  use  of 
federal  investment  in  R&D  and  mandates  that  transfer  of  technology  to  the  private  sector 
when  appropriate.  This  act  also  established  the  Offices  of  Research  and  Technology 
Applications  (ORTAs)  within  the  federal  laboratories  to  disseminate  information  about 
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federal  products,  processes,  and  services  (38:  23;  2).  The  Federal  Technology  Transfer 
Act  (FTTA)  of  1986  laid  the  foundation  for  Cooperative  Research  and  Development 
Agreements  (CRD As)  betAveen  federal  laboratories  and  private  enterprises  (38: 23,24). 
This  act  gives  government  agencies  the  authority  to  conduct  cooperative  research  with 
outside  parties  and  negotiate  patent  licenses.  In  addition,  the  FTTA  established  the 
Federal  Laboratory  Consortium  (FLC)  which  supports  and  helps  laboratories  'with 
transfer  activities  (38:  23;  2).  The  National  Competitiveness  Technology  Transfer  Act  of 
1989  was  implemented  to  aid  in  authorizing  CRD  As  for  Govemment-o-wned,  contractor- 
operated  (GOCO)  organizations.  This  act  also  protects  trade  secret  information 
developed  under  a  CRDA  from  disclosure  under  the  Freedom  of  Information  Act.  These 
policies  and  additional  directives  have  provided  a  foundation  for  successful  transfer 
activities  (38:  24;  2).  In  spite  of  mechanisms  being  in  place  for  successful  technology 
transfer,  the  federal  government  and  private  industry  must  still  work  to  improve  the 
effectiveness  of  their  transfer  activities  (38:  28). 

Despite  the  legislative  attention  given  to  technology  transfer  and  the  commitment 
of  government  agencies  to  make  it  work,  successful  commercialization  only  happens 
when  and  if  technology  translates  into  a  profitable  product  or  process  with  an  economic 
benefit  (38:  28).  From  a  business  point  of  -view,  certain  basic  steps  are  required  for  a 
successful  transfer.  They  include  identifying  a  need,  sourcing  the  technology,  assessing 
the  technology,  acquiring  the  technology,  financing  the  project,  transferring  the 
technology,  implementing  the  technology,  and  eventually  terminating  the  transfer 
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(38:  24).  Technology  transfers  that  do  not  meet  the  above  criteria  are  likely  to  be 
unsuccessful  and  could  waste  valuable  resources  in  the  public  and  private  sector. 

Technology  transfer  commercialization  success  can  not  be  the  sole  measure  of 
the  government’s  technology  transfer  success  because  many  activities  are  out  of  the 
government’s  span  of  control  (16:1 1) .  AFMC  has  a  well  defined,  hierarchical  transfer 
management  process  in  place.  The  management  process  includes  the  following  six  steps: 

a.  Develop  transfer  strategy 

b.  Identify  technologies 

c.  Market  the  technologies 

d.  Develop  the  transfer  agreement 

e.  Transfer  the  technology 

f  Post  transfer  administration 

These  steps  are  at  a  command  (macro)  level.  At  the  laboratory  or  product  center  level,  the 
process  has  many  steps  which  fall  within  the  six  steps  of  the  command  process  (32:  6;  2). 
In  other  words,  the  process  can  be  tailored  to  meet  the  needs  of  each  given  organizational 
level.  These  differences  provide  each  laboratory  or  center  the  flexibility  required  to 
efficiently  carry  out  the  transfer  process  as  they  see  fit.  The  logic  behind  this  is  that  it 
maximizes  resources  at  every  level  and  leads  to  effective  technology  transfer  activities. 
There  is  a  sound  process  in  place,  but  measurement  of  the  process  has  been  a  difficult 
task.  The  goal  at  AFMC  is  to  develop  measures  at  the  command  level  that  will  indicate 
the  success  of  technology  transfer  activities  in  the  labs  and  centers. 

Technology  Transfer  Measurements 

Technology  transfer  has  been  defined  as  the  managed  process  of  conveying  a 
technology  from  one  party  to  its  adoption  by  another  (34:  65).  There  are  three  distinct 
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roles  within  the  technology  transfer  process:  sponsor,  developer,  and  adopter.  Sponsors 
fund  technology  development  and  disseminate  information  about  government-sponsored 
technology  to  the  private  sector  and  facilitate  transfer.  Developers  develop  or  apply 
technology  imder  government  or  private  sponsorship  and  funding.  Adopters  are  the  users 
or  potential  users  of  government  technology  (34:  66-68).  Within  the  three  roles,  the 
success  of  technology  transfer  has  been  difficult  to  measure.  Some  of  the  most  common 
measurements  of  technology  transfer  are  included  below: 

a.  The  attainment  of  transfer  objectives. 

b.  Changes  in  the  user’s  revenues  or  costs,  resulting  from  the  transfer. 

c.  The  number  of  products  launched. 

d.  The  degree  of  technology  adoption  or  rejection. 

e.  The  degree  to  which  a  significant  emotional  and  financial  commitment  is 
made  to  the  routine  use  of  the  technology  (34:  70). 

These  measurements  are  certainly  useful,  but  their  focus  is  primarily  on  the 
success  of  the  transfer  in  terms  of  the  adopter’s  objectives.  It  is  difficult  to  make 
inferences  about  the  success  of  developer’s  and  sponsor’s  transfer  efforts  based  on  these 
common  measurements  of  technology  transfer  activities.  These  measures  could  be 
classified  as  firm  level  measurements  because  they  are  more  applicable  to  the  private  firm 
or  company  that  adopts  the  technology.  Some  more  specific  firm  level  technology 
transfer  performance  measurements  are  relevant.  They  include:  number  of  new  products 
developed,  number  of  patents  filed,  and  accounting  measures  of  financial  performance 
such  as  return  on  investment  (34:  70-72).  Unfortunately,  these  measures  fail  to  address 
the  needs  of  the  government  which  seeks  valid  performance  measurement  of  the  transfer 
process  itself. 
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Two  important  areas  of  concern  within  the  government  are  the  measurement  of 
intangible  benefits  such  as  goodwill  within  the  commvmity,  the  cost/work  avoidance 
arising  from  transfer  activities,  and  the  measurement  of  economic  costs  associated  to 
trade-offs  made  by  federal  laboratories  in  the  use  of  funds,  human  resources,  and  time 
(33:21-23).  The  government  has  been  unable  to  develop  and  document  a  valid 
methodology  for  quantifying  the  tangible  and  intangible  benefits  of  its  transfer  efforts. 

The  consensus  is  that  these  benefits  are  critical  to  the  success  of  transfer  activities  and 
should  be  weighed  heavily  when  building  a  transfer  strategy  (33:23-27).  However,  there 
are  very  few  decision  makers  in  any  business  that  feel  comfortable  measuring  the  success 
of  their  programs  or  policies  using  a  decision  support  system  based  on  unquantified, 
intangible  benefits  (25: 26;  37:35).  To  accurately  measure  the  success  of  government 
transfer  activities,  a  model  must  be  developed  that  combines  the  tangible  revenues 
received  by  the  government  with  the  intangible  benefits  (revenues  or  costs  avoided).  This 
will  lead  to  a  more  accurate  measure  of  transfer  effectiveness  (17:  7). 

Many  existing  models  fail  to  adequately  measure  transfer  activities  because  of  the 
inherent  difficulty  associated  with  capturing  transfer  outcomes.  First  of  all,  in  the  case  of 
revenues  received  (transfer  payments,  licensing  and  royalty  fees)  outcomes  are  often 
delayed  by  long  development  or  payback  times.  This  leads  to  difficulties  since  revenues 
can  rarely  be  linked  to  specific  transfers  (17: 4).  Therefore,  it  is  difficult  to  link  the 
results  of  a  given  transfer  to  the  transfer  itself,  making  it  impossible  to  benchmark  the 
process  used  in  successful  transfers  to  the  process  used  in  less  successful  transfers. 
Secondly,  the  different  organizations  involved  with  the  transfer  may  have  different 


transfer  goals  and  measurements  of  success.  Finally,  the  commercial  success  of  a  given 
transfer  depends  on  factors  other  than  a  successful  transfer.  These  differences  contribute 
to  the  difficulties  in  measuring  transfer  progress,  success,  and  overall  effectiveness  (17: 
3-6;  33:  23). 

Current  measurement  within  AFMC  focuses  on  the  potential  socio-economic 
benefits  derived  from  transfer  activities.  The  command  is  measuring  the  number  of 
transfer  agreements  by  fiscal  year  and  the  total  dollars  invested  by  the  government  and 
outside  partners  that  can  be  linked  to  transfer  agreements.  The  logic  in  using  these 
measures  is  that  increases  in  agreements  and  investment  will  likely  result  in  increased 
potential  positive  impacts  in  the  form  of  socio-economic  benefits  (17:  3-6).  The  current 
trend  is  that  both  agreements  and  investment  are  increasing  within  AFMC.  This  indicates 
that  the  technology  transfer  program  is  working  and  providing  socio-economic  benefits. 
Another  measure  of  transfer  success  is  the  amoimt  of  revenue  received  by  AFMC  in  the 
form  of  royalty  payments  and  licensing  fees.  These  revenues  represent  increased  value  to 
the  outside  partner  that  can  be  attributed  to  the  transfer.  Socio-economic  benefits  will 
increase  as  the  worth  of  the  transfer  grows  to  the  outside  partner.  Unfortunately,  this 
relationship  is  difficult  to  measure  because  the  outside  partner,  not  the  Air  Force,  owns 
the  data  (17:  3-6). 

By  taking  a  closer  look  at  two  common  measures  of  transfer  effectiveness, 
number  of  licenses  granted  and  the  royalty  income  generated,  it  is  easy  to  see  why 
additional  measures  should  be  considered.  Simply  looking  at  the  number  of  licenses 
granted  gives  no  indication  of  the  impact  or  benefits  derived  from  the  transfer  and  the 
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royalty  payments  usually  lag  behind  the  licensing  activity,  thus  providing  a  tardy  measure 
of  transfer  success  (33:  20-23). 

Strategy  plays  a  key  role  in  the  measures  used  to  assess  the  effectiveness  of 
transfer  activities.  Technology  push  and  market  pull  strategies  result  in  two  distinct  types 
of  performance  measurements.  Technology  push  strategy  effectiveness  can  be  measured 
by  the  number  of  licenses,  site  visits,  technical  briefs  or  papers,  technical  presentations, 
time  spent  on  transfer  activities,  transfer  budgets,  and  transfer  expenditures.  These 
measures  indicate  the  cost  of  transfer  activities  and  the  level  of  effort  put  forth  by  the 
sponsor  or  developer  of  a  given  technology.  They  are  short  term  measures  of 
effectiveness,  while  technology  pull  metrics  are  long  term  measures  of  effectiveness. 
Market  pull  measures  include  competitive  advantage  gains,  cost  savings,  jobs  created, 
market  share  gains,  new  businesses  started,  commercial  sales,  new  products,  productivity 
gains,  royalties,  return  on  investment,  success  stories,  technical  problems  solved,  and  user 
satisfaction.  These  metrics  are  long  term  measurements  that  indicate  the  success  of 
adopters  in  utilizing  the  technology  after  it  has  been  transferred  (33: 24,25).  Market  pull 
measures  are  out  of  the  government’s  span  of  control  and  therefore,  may  not  be  an 
accurate  measure  of  government  technology  transfer  effectiveness  (16:  8).  These 
measures  are  useful  in  measuring  the  effort  the  government  puts  into  transfer  activities, 
but  there  is  no  methodology  in  place  to  measure  the  specific  returns  or  benefits  received 
by  the  government  as  a  result  of  it’s  effort. 

Given  that  data  is  available,  a  cost/benefit  analysis  model  is  a  logical  choice  for  a 
methodology  to  be  used  by  the  government  to  assess  the  effectiveness  of  its  transfer 
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activities.  Measuring  the  costs  or  effort  put  forth  in  transfer  activities  (technology  push 
metrics)  is  not  difficult,  but  the  data  on  the  benefits  of  technology  transfer  is  limited  and 
not  readily  available.  Items  such  as  revenues  received  in  the  form  of  license  fees,  royalty 
payments,  or  leasing  fees  represent  cash  payments  to  the  government  that  can  be  used  to 
formulate  a  discounted  cash  flows  return  on  investment  model  that  measures  the 
effectiveness  of  government  transfer  activities.  Although  these  revenue  streams  are 
important,  there  are  many  intangible  benefits  received  by  the  government  as  a  result  of  its 
transfer  activities.  These  intangible  benefits  which  usually  exceed  the  benefits  of  cash 
revenues  generated  by  transfer  activities  are  difficult  to  quantify.  Therefore,  they  are 
often  overlooked  by  senior  decision  makers.  To  get  a  true  measure  of  technology  transfer 
success,  these  intangible  benefits  must  be  analyzed  by  decision  makers. 

Measuring  Return  on  Investment  (ROD 

The  four  most  commonly  used  methodologies  for  ranking  projects  based  on  their 
projected  cash  flows  are  payback  period,  accounting  rate  of  return  (ARR),  net  present 
value  (NPV),  and  internal  rate  of  return  (IRR)  (23: 131-135;  6:  384).  This  section  will 
explain  each  methodology  and  examine  strengths  and  weaknesses  of  each  methodology. 
Once  an  acceptable  methodology  is  identified,  it  will  be  incorporated  into  Chapter  Three 
and  used  to  calculate  the  ROI  for  the  transfers  highlighted  in  Chapter  Four. 

The  payback  period  is  probably  the  simplest  and  quickest  ROI  methodology 
available  to  the  decision  maker.  It  is  simply  the  expected  number  of  years  required  to 
recover  the  projects  original  cost.  It  is  calculated  by  estimating  a  project’s  initial  cost  and 
its  projected  cash  flows.  The  cash  flows  are  then  subtracted  from  the  original  investment 
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until  the  cumulative  total  value  reaches  zero;  the  point  where  the  project  has  paid  for 
itself  (6:  389-391).  The  example  below  provides  a  quick  glance  at  how  the  payback 


period  would  be  calculated: 

Both  project  X  and  Y  have  the  same  initial  jirstyear  cost  ($700. 00)  but  project  Y  could  be 
paid  off  in  only  three  years  while  project  X  would  not  be  paid  off  until  the  beginning  of 
the  fifth  year.  Based  on  the  payback  period  methodology  project  Y  would  be  preferable  to 
project  X,  but  whether  Y  would  be  accepted  over  Y  would  depend  on  the  accepted 
payback  period. 


YEAR 

Project  X 

Cumulative  X 

Project  Y 

Cumulative  Y 

0 

$(700.00) 

$ 

(700.00) 

$(700.00) 

$ 

(700.00) 

1 

100 

$ 

(600.00) 

300 

$ 

(400.00) 

2 

50 

$ 

(550.00) 

200 

$ 

(200.00) 

3 

300 

$ 

(250.00) 

200 

$ 

- 

4 

200 

$ 

(50.00) 

100 

$ 

100.00 

5 

800 

$ 

750.00 

50 

$ 

150.00 

Total  CF 

$  1,450 

$  850 

FIGURE  2.1  Project  X  and  Y  Payback  Period  Example 
The  payback  period  is  most  useful  as  a  break-even  analysis  tool.  It  provides  the 
decision  maker  with  an  idea  of  how  long  funds  will  be  tied  up  in  the  project.  It  is  a  good 
method  for  evaluating  the  liquidity  of  two  competing  projects.  Unfortunately,  the 
payback  period  does  not  take  the  time  value  of  money  into  consideration  and  it  does  not 


consider  cash  flows  that  occur  after  the  payback  period  is  over  (6:  390).  For  instance,  in 
the  above  example  project  X  had  an  expected  $750.00  positive  cash  flow  at  the  end  of 
year  five  which  was  greater  than  the  expected  $150.00  cash  flow  provided  by  project  Y. 
By  only  using  the  payback  period,  project  X,  the  more  profitable  alternative  would  have 


been  rejected. 
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The  ARR  focuses  on  a  project’s  contribution  to  a  firm’s  net  income  rather  than 

cash  flows.  This  methodology  takes  depreciation  into  consideration  when  evaluating  an 

investment  project.  ARR  is  measured  as  a  ratio  of  a  project’s  average  annual  net  income 

to  its  average  investment  (6;  391).  See  the  example  below  for  a  detailed  explanation; 

calculated  using  straight  line  5  yr.  depreciation  and  zero  salvage  value: 

Average  annual  income  =  Average  cash  flow  -  Average  annual  depreciation 

Project  X  avg.  annual  income  =  ($  1450/5)  -  ($700/5) 

=  $290  -  140 
=  $150 

Project  Y  avg.  annual  income  =  ($  850/5)  -  ($  700/5) 

=  $170  -  140 
=  $30 

Average  investment  =  (Cost  +  Salvage  value)  /  2 

Avg.  investment  for  both  projects  =  ($700  +  0)/2=$  350 

ARR  =  Average  annual  income  /  Average  investment 

ARR  for  Project  X  =  $150/ $350  =  42.8% 

ARR  for  Project  Y  =  $  30  /  $350  =  8.5% 

Based  on  this  example,  Project  X  is  the  best  alternative  because  it  has  a  much  higher 
ARR  at  42.8%.  The  ARR  does  address  cash  flows  that  occur  after  the  payback  period. 
However,  it  fails  to  account  for  the  time  value  of  money  and  therefore,  could  lead  to  poor 
business  decisions.  For  this  reason,  the  NPV  and  IRR  methodologies  are  more  effective 
for  business  decisions  (6:  392;  23: 133). 

The  net  present  value  calculation  is  a  popular  return  on  investment  (ROI) 
methodology  used  by  decision  makers  in  deciding  between  two  alternative  projects.  A 
project’s  net  present  value  is  calculated  by  comparing  the  discounted  cash  flows  of  a 
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project  to  the  costs  associated  with  a  project  to  determine  the  project’s  real  value.  The 
cash  flows  are  discounted  at  the  company’s  cost  of  capital  (the  interest  rate  they  could 
have  earned  by  making  a  low  risk  investment)  to  account  for  the  time  value  of  money. 
Projects  with  discounted  cash  flows  (revenues)  greater  than  the  initial  cash  outlay  (project 
cost)  are  acceptable  and  are  deemed  good  business  decisions  (6:  394;  23:  133-134).  The 
greater  the  project’s  net  present  value,  the  higher  its  return  on  investment  and  therefore, 
the  more  desirable  it  is.  The  tables  below  provide  an  example  of  how  NPV  is  calculated 


and  how  it  can  be  used  to  decide  between  two  alternative  projects: 

for  this  example  assume: 
k  =  10%  interest  rate  (the  firm’s  cost  of  capital) 
t  =  The  year  cash  flow  was  produced 

Project  1  Net  Present  Value  Calculation  (Return  On  Investment) 

Initial  Cost  year1  year  2  year  3  year  4  Total  Cash  flow 
$  (450.00)  $125.00  $125.00  $125.00  $200.00  $  575.00 

(1  +  k)*t  (l  +  kj'^t  (1  +  k)*t  (l  +  kj'^t 

$  113.64 
$  103.31 
$  93.91 

$  136.60 
$  (2.54)  NPV 


Project  2  Net  Present  Value  Calculation  (Return  On  Investment) 

Initial  Cost  year1  year  2  year  3  year  4  Total  Cash  flow 
$  (450.00)  $200.00  $225.00  $100.00  $  50.00  $  575.00 

(1+k)At  (l  +  kj'^t  (1  +  k)*t  (1  +  k)*t 

$  181.82 
$  185.95 
$  75.13 

$  34.15 

$  27.05  NPV 


FIGURE  2.2  Project  1  and  2  Net  Present  Value  Example 
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In  this  short  example,  project  2  has  a  higher  net  present  value  because  of  its  large  cash 
flows  in  years  1  and  2.  These  cash  flows  minimize  the  impact  of  discounting  or  the  time 
value  of  money  and  makes  project  2  acceptable,  whereas  project  1  would  be  rejected. 

The  internal  rate  of  return  is  another  popular  ROI  methodology  that  can  be  used  to 
analyze  a  project’s  value  to  the  firm  based  on  discounted  cash  flows.  The  internal  rate  of 
return  (IRR)  is  the  interest  rate  that  causes  a  project’s  net  present  value  to  be  zero.  The 
higher  the  project’s  IRR,  the  more  desirable  it  is.  Ideally,  the  IRR  should  be  higher  than 
the  firm’s  cost  of  capital  for  the  project  to  be  considered  a  success  (6:  396;  23: 134).  IRR 
is  often  used  by  decision  makers  in  the  business  world  because  it  is  easy  to  understand 
and  it  is  easily  compared  to  alternative  uses  of  capital.  The  IRR  for  a  project  can  be 
easily  compared  to  the  interest  rate  earned  on  an  alternative  investment  or  a  given 
industry’s  hurdle  rate.  For  example,  if  a  project  has  an  initial  cost  of  $1 0,000  and 
provides  an  IRR  of  1 1%,  it  would  be  less  desirable  than  the  alternative  of  investing  that 
$10,000  into  a  mutual  ftmd  that  was  currently  showing  a  15%  return  on  investment. 

The  NPV  and  IRR  are  both  useful,  sound  decision  criteria  for  analyzing  a 
project’s  worth  to  the  firm  or  organization.  In  order  to  calculate  the  IRR  or  NPV,  the 
following  steps  must  take  place: 

a.  The  cost  of  the  project  must  be  determined. 

b.  The  cash  flows  and  revenues  from  a  project  must  be  obtained  or  accurately 
estimated.  This  would  include  revenues  linked  to  the  intangible  benefits  of  a 
project. 

c.  The  riskiness  of  a  project’s  cash  flows  must  be  estimated  to  develop 
probability  distributions  for  the  expected  revenues  or  cash  flows. 

d.  Management  determines  the  appropriate  cost  of  capital  for  a  project  based  on 
the  riskiness  of  that  project’s  expected  returns  or  revenues. 

e.  Expected  cash  flows  are  discounted  to  obtain  the  project’s  NPV  (6:  387). 


These  steps  are  not  difficult  to  understand  or  implement.  The  difficult  aspect  of 
using  a  conventional  ROI  methodology  to  evaluate  technology  transfer  activities  is  that 
developing  estimates  of  a  project’s  actual  or  projected  revenues  and  benefits  can  be  an 
extremely  difficult  and  cumbersome  task.  The  reason  for  this  difficulty  is  that  a  transfer’s 
intangible  benefits  are  sometimes  impossible  to  quantify  and  therefore,  can  not  be  used  in 
the  traditional  ROI  methodologies.  When  decision  makers  can  not  see  the  actual  benefits 
of  a  project  in  terms  of  dollars,  they  may  find  it  difficult  to  justify  investments  in  that 
project.  This  could  lead  to  poor  business  decisions  when  intangible  benefits  are  not 
considered  and  their  contribution  would  cause  the  project’s  benefits  to  outweigh  the 
project’s  costs.  Another  problematic  scenario  is  that  sometimes  intangible  benefits  are 
considered  as  a  decision  criteria,  but  are  overstated,  understated,  or  not  quantified.  The 
intangible  benefits,  which  can  be  very  subjective,  can  cause  decision  makers  to  be  overly 
optimistic  about  a  project’s  returns  and  could  lead  to  a  project  being  accepted  when  it 
should  be  rejected  (37:  35,36). 

The  nature  of  intangible  benefits  is  that  they  often  appear  or  effect  areas  in  an 
organization  that  decisionmakers  never  accounted  for  when  making  the  initial  decision  on 
a  project.  Many  times,  intangible  benefits,  when  realized,  are  not  attributed  to  the 
project  that  created  them  in  the  first  place  (25:  27).  This  situation  could  potentially  cause 
a  misallocation  of  benefits  or  revenues  to  a  given  project.  The  need  to  identify  and  define 
intangible  benefits  is  critical  to  the  evaluation  of  a  project  like  technology  transfer 
activities.  This  can  be  viewed  as  a  two  part  process.  First,  the  intangible  benefits,  once 
identified,  should  be  put  into  some  quantifiable  form.  Second,  the  magnitude  of  the  value 
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must  be  specified  (25:  27).  Once  these  two  steps  are  accomplished,  probability 
distributions  can  be  assigned  to  the  intangible  benefits  of  a  project.  The  use  of 
probability  distributions  are  necessary  when  calculating  technology  transfer  ROI  because 
by  nature  the  intangible  benefits  used  in  the  ROI  calculation  can  never  be  known  with 
certainty.  The  magnitude  of  the  intangible  benefits  are  usually  an  unknown  and  analysts . 
are  often  uncertain  about  whether  or  not  the  intangible  benefits  will  materialize. 

Existing  Methodologies 

There  have  been  no  published  methodologies  used  to  quantify  the  intangible 
benefits  of  technology  transfer  activities.  However,  the  information  technology  (IT) 
arena  faces  many  of  the  same  difficulties  as  technology  transfer  activities.  IT  investments 
are  usually  beneficial  to  the  firm  and  increase  productivity,  but  many  of  the  benefits 
derived  from  IT  investment  are  intangible  and  difficult  to  quantify  (30:  45). 
Decisionmakers  have  been  coping  with  the  problem  of  IT  investment  justification  for 
many  years.  As  a  result,  there  are  several  published  methodologies  that  have  been  used 
to  quantify  the  intangible  benefits  of  IT  investment.  These  methodologies  could 
potentially  be  utilized  to  quantify  the  benefits  of  technology  transfer  activities. 

Oracle  Corp.  has  developed  a  model  called  CB-90  which  takes  decisionmakers 
through  a  step-by-step  approach  to  analyzing  an  IT  investment  decision.  This  three  step 
approach  requires  decision  makers  to  look  at  the  tangible  cost/benefit  analysis  of 
competing  projects,  the  intangible  benefit  analysis  of  competing  projects,  and  the  risks 
associated  with  those  intangible  benefits  (30: 45).  The  first  part  of  the  CB-90  model  is 
the  tangible  cost/benefit  analysis  where  the  NPV  and  IRR  are  used  to  evaluate  competing 
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IT  investment  projects  against  the  status  quo.  The  tangible  and  intangible  benefits  are 
identified  and  agreed  upon  by  a  group  of  decision  makers.  Once  this  is  accomplished, 
weights  are  assigned  to  the  tangible  and  intangible  benefits  and  the  intangible  risks.  For 
example  tangible  benefits  might  be  given  a  60%  weight,  intangible  benefits  a  15% 
weight,  and  intangible  risks  a  25%  weight.  Each  benefit  and  risk  is  then  rated  on  a  scale 
of  1  -5  as  to  the  likelihood  of  the  proposed  outcome.  The  weight  factor  is  then  multiplied 
by  the  rate  to  arrive  at  a  score  for  each  option.  The  investment  decision  with  the  highest 
score  is  selected  (30;  46).  The  example  below  demonstrates  the  CB-90  model: 


In  this  hypothetical  situation,  options  1  and  2  are  competing  projects.  Their  tangible  and 
intangible  benefits  and  risks  have  been  identified  and  computed  using  the  NPV  method. 
Weights  were  assigned  to  each  of  the  project ’s  benefits  and  risks.  The  rates  represent  the 
likelihood  of  each  outcome  occurring.  Option  2  has  the  higher  total  score  and  would  be 
preferred  over  option  1. 


Option  1 

Option  2 

Tangible  Benefits 

Weight 

Rate 

Score 

Rate 

Score 

Cost  Savings 

30% 

2.2 

0.66 

2.7 

0.81 

Other  Tangibles 

35% 

2.2 

0.77 

2.7 

0.945 

Total  Tangible  Score 

65% 

1.43 

1.755 

Intangible  Benefits 

Flexible  Budgets 

5% 

2 

0.1 

4 

0.2 

Improved  Reporting 

5% 

2 

0.1 

5 

0.25 

Improved  Analysis 

5% 

4 

0.2 

5 

0.25 

Productivity  Increases 

5% 

3 

0.15 

4 

0.2 

Subtotal  intangible  score 

20% 

0.55 

0.9 

Intangible  Risks 

Poor  Integration 

3% 

-2 

-0.06 

-2 

-0.06 

Resistance  to  Change 

4% 

-2 

-0.08 

-1 

-0.04 

Incomplete  Implementation 

5% 

-4 

-0.2 

-2 

-0.1 

Losing  Employees 

3% 

-1 

-0.03 

-1 

-0.03 

Subtotal  Intangible  Risks  Score 

15% 

-0.37 

-0.23 

Total  Benefits 

100% 

1.61 

2.43 

FIGURE  2.3  CB  -  90  Model  Example 
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This  decision  making  model  is  suspect  due  to  the  amount  of  subjectivity  in  the 
development  of  weights  and  ratings.  The  weights  and  rates  were  developed  by  a 
committee  of  decision  makers.  Unfortunately,  there  is  no  explanation  of  the 
methodologies  used  to  develop  these  factors.  CB-90,  although  suspect,  is  a  step  in  the 
right  direction  in  that  it  requires  decisionmakers  to  identify  and  discuss  the  potential 
intangible  benefits  and  risks  associated  with  a  given  project.  In  addition,  CB-90  does 
offer  a  structured  approach  in  the  form  of  an  investment  decision  matrix  which  can  be 
easily  understood  and  used  by  decision  makers  (30: 47).  In  order  to  make  this  model 
more  usefiil,  formal  risk  analysis  techniques  could  be  used  to  develop  the  rates  for  each 
outcome. 

In  his  research,  Mathias  Schumann  (1989)  develops  a  similar,  but  more  in-depth 
approach  to  quantifying  the  intangible  benefits  of  office  automation.  Shumann  explains 
that  an  office  automation  system  can  cause  a  shift  towards  higher  valued  work  activities 
as  well  as  a  reduction  in  non-productive  work  time  (28:  21).  Office  automation  is 
classified  into  three  categories: 

a.  Substitutive  applications  replace  manual  process  and  lead  to  the  most  obvious 
benefits; 

b.  Complementary  applications  -  offer  additional  support  functions  and  tools  to 
make  workers  more  productive,  but  the  benefits  are  difficult  to  identify  and 
quantify; 

c.  Innovative  applications  -  yield  competitive  advantages  and  result  in  increased 
revenues. 
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The  major  source  of  savings  or  benefits  is  the  increase  in  productivity  associated  with 
automation.  The  model  documented  by  Schumann  attempts  to  bring  this  out  (28:  22-25). 
The  following  five  step  framework  was  used  to  build  the  model: 

1.  Employees  were  grouped  into  different  categories  based  on  their  activities. 

-  Managers  -  top  and  middle  management 

-  Professionals  -  technical,  administrative,  and  scientific 

-  Administrative 

-  Clerical; 

2.  Decomposition  of  work  activities  at  each  level  and  the  percentage  of  time 
spent  on  each  activity  was  determined; 

3.  Evaluation  of  the  task  breakdown  of  the  different  activities  for  each  employee; 

4.  Estimation  of  the  office  automation  savings  potential  related  to  these  tasks; 

5.  Derivation  of  possible  time  savings,  productivity  improvements,  and  total 
savings  quantified  with  labor  costs. 


A  sample  work  profile  was  developed  that  showed  the  percentage  of  a  worker’s  total 
work  that  was  taken  up  by  each  individual  work  task  performed  by  that  worker.  Another 
table  was  developed  to  display  the  potential  work  savings  over  a  given  year  for  middle 
managers.  This  work  savings  was  calculated  as  the  decrease  in  percentage  of  time  to 
perform  a  given  work  task  that  resulted  from  office  automation.  For  example,  office 
automation  may  enable  a  middle  manager  to  reduce  the  amount  of  time  used  to  handle 
documents  by  45%.  If  that  task  (document  handling)  made  up  5%  of  the  managers  total 
work  day,  then  it  would  be  reduced  by  2.25%,  leaving  him  or  her  additional  time  to  spend 
on  other  work  tasks.  The  table  follows  that  logic  throughout  and  results  in  a  total  percent 
work  savings  resulting  from  office  automation.  This  total  percentage  is  then  multiplied 
by  a  dollar  amount  (e.s.,  worker’s  salary)  to  obtain  an  annual  cost  valuation  (or  dollar 
savings).  The  results  of  the  table  are  broken  down  by  triangular  distribution  parameters 
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(pessimistic,  most  likely,  and  optimistic),  but  there  is  no  indication  that  any  formal  risk 
analysis  was  done  to  arrive  at  the  outcomes. 

Because  the  study  is  focused  at  the  individual  worker  level  of  the  organization,  the 
results  do  not  directly  apply  to  the  needs  of  AFMC  and  the  Technology  Transfer  Office. 
However,  some  of  the  tools  used  can  be  applied  at  a  higher  organizational  level.  The 
same  methods  used  to  decompose  the  work  into  specific  tasks  can  be  applied  when  trying 
to  determine  the  types  of  intangible  benefits  that  can  be  applied  to  technology  transfer 
activities.  The  study  conducted  by  Shumann  is  another  step  in  the  right  direction,  but 
would  have  to  be  modified  to  be  applicable  to  AFMC. 

Another  methodology  described  by  Richard  Pastore  discusses  the  need  to  identify 
tangible  and  intangible  benefits  and  ensure  that  they  align  with  the  organization’s  goals 
and  objectives.  Information  Economics  (IE)  is  a  system  of  quantifying  intangible 
benefits  of  a  project  and  ranking  the  projects  based  on  their  expected  contribution  to 
business  objectives  (24:  66). 

The  first  step  in  this  process  is  for  top  executives  to  identify  corporate  objectives 
and  give  them  a  relative  weight  based  on  their  importance.  The  second  step  in  the 
process  is  that  proposed  projects  are  given  a  score  based  on  their  estimated  impact  to  each 
objective.  In  this  system,  ROI  still  receives  a  significant  weight,  but  it  is  no  longer  the 
only  criteria  for  evaluating  alternative  projects.  The  third  step  in  the  process  is  a  peer 
review,  where  each  alternative  project  proposal  is  brought  to  the  table  with  scores  filled 
in.  The  projects  that  align  closely  with  the  corporate  strategy,  goals,  and  objectives  will 
most  likely  be  accepted  and  projects  that  do  not  fit  into  the  corporate  strategy  will  be 
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rejected.  IE  provides  a  common  sense,  team  approach  to  evaluating  projects  based  on 
criteria  other  than  ROI  (24:  67-70). 

The  IE  approach  also  has  some  flaws  that  may  make  it  difficult  to  implement 
within  the  DoD.  IE  can  be  a  difficult  concept  to  grasp.  In  addition,  it  can  take  a  great 
deal  of  time  for  managers  to  go  through  the  IE  process  when  looking  at  alternative 
projects.  Because  IE  places  a  strong  emphasis  on  linking  projects  to  the  organization’s 
strategic  plan,  involvement  by  senior  executives  is  vital.  Therefore,  IE  can  be  a  costly 
methodology  to  implement.  The  biggest  drawback  to  IE  is  its  lack  of  a  system  to 
measure  post-implementation  benefits.  Finally,  IE  is  not  a  hard  science  and  all  this  work 
may  provide  very  little  solid  justification  or  hard  numbers  to  support  management 
decisions  (24:  73). 

Probability  encoding  is  an  interview  process  used  to  extract  and  quantify  an 
individual’s  judgment  about  certain  quantities  or  outcomes.  Probability  encoding  is 
commonly  used  during  the  probabilistic  phase  of  decision  analysis  where  uncertainty  can 
be  incorporated  into  the  analysis  by  assigning  probability  distributions  to  important 
outcomes  or  variables.  By  using  probability  encoding,  the  risks  involved  with 
quantifying  intangible  benefits  can  be  accounted  for  (35:  340-342).  Estimates  of  a 
transfer’s  intangible  benefits  can  be  made  using  expert  judgment  or  opinion.  Once  these 
numbers  are  provided,  a  probability  distribution  can  be  built  for  the  estimates  and  these 
estimates,  once  quantified,  can  be  added  to  the  ROI  calculation. 
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Probability  encoding  is  a  structured  interview  process.  The  following  guidelines 
apply  when  attempting  to  encode  uncertain  quantities: 

a.  Choose  only  uncertain  quantities  that  are  important  to  the  decision.  There  is 
no  sense  encoding  the  quantity  of  an  intangible  benefit  that  does  not  have  a 
major  impact  to  the  organization; 

b.  Define  the  quantity  as  an  unambiguous  state  variable; 

c.  Structure  the  quantity  carefully  so  that  it  is  imderstandable  to  the  interviewee 
and  is  appropriate  for  cost  benefit  analysis; 

d.  Clearly  define  the  quantity.  Precision  is  critical  to  probability  encoding; 

e.  Describe  the  quantity  using  a  scale  that  is  meaningful  to  the  subject. 

“By  following  these  guidelines,  reasonable  estimates  and  distributions  can  be  built  for 
uncertain  quantities  (35:  343).”  These  estimates  can  then  be  used  in  decision  making 
models  or  ROI  calculations. 

One  difficult  aspect  of  probability  encoding  is  that  it  is  a  process  that  relies  on 
human  judgment.  In  many  cases,  judgment  can  be  impaired  by  individual  bias.  Biases 
can  take  one  of  two  different  forms.  Displacement  bias  is  characterized  by  a  shift  in  the 
distribution  either  upward  or  downward,  while  variability  bias  results  in  a  change  in  the 
shape  of  the  distribution  itself  For  example,  the  tendency  of  an  expert  to  estimate  high 
and  low  values  unrealistically  close  to  the  estimated  mean  would  be  a  form  of  variability 
bias.  This  situation  would  result  in  a  tighter  distribution.  The  interviewer  must  always 
guard  against  bias  and  take  the  necessary  steps  to  prevent  it  when  possible  (35:  345;  4: 
64-66). 

Most  encoding  methods  are  based  on  questions  that  lead  to  answers  that  can  be 
expressed  as  part  of  a  cumulative  density  function.  There  are  three  basic  types  of 


encoding  methods.  The  P-methods  require  the  subject  to  respond  by  specifying  points  on 
a  probability  scale  while  the  values  remain  fixed.  The  V-methods  require  the  subject  to 
specify  points  on  the  value  scale  while  the  probabilities  remain  fixed.  Finally,  PV- 
methods  require  the  subject  to  respond  to  both  scales  jointly.  The  type  of  method  used 
varies  according  to  the  outcome  or  variable  that  is  being  quantified  (35:  350). 

The  probability  encoding  technique  can  easily  be  integrated  into  any  of  the  above 
mentioned  methodologies  to  make  them  more  robust.  For  example,  it  could  easily  be 
incorporated  into  the  CB-90  model  to  replace  the  rate  factor  calculations.  By 
incorporating  probability  encoding  and  some  basic  risk  analysis  techniques  into  the  CB- 
90  model,  a  robust  model  could  be  developed  to  quantify  the  intangible  benefits  of 
technology  transfer  activities.  The  result  of  this  methodology  would  be  a  ROI  calculation 
similar  to  the  NPV  example,  but  the  NPV  calculation  could  be  stated  as  a  confidence 
interval  with  a  range  of  likely  outcomes.  This  methodology  would  provide  decision 
makers  with  a  rational  decision  making  tool  and  would  allow  them  to  compare  projects 
that  contain  tangible  and  intangible  benefits. 

Summary 

This  chapter  focused  on  the  current  technology  transfer  and  return  on  investment 
literature.  Specific  areas  addressed  were  technology  transfer  policies,  current 
measurements  of  transfer  success,  ROI  measurements,  and  existing  methodologies  used 
to  quantify  tangible  and  intangible  benefits.  The  goal  of  this  research  is  to  build  an 
effective  methodology  that  will  enable  decision  makers  to  analyze  the  benefits  of 
technology  transfer  activities.  Chapter  Three  will  focus  on  building  that  methodology. 
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III.  Methodology 

Overview 

After  reviewing  the  literature,  it  is  clear  that  a  more  effective  methodology  is 
necessary  to  measure  the  return  on  the  government’s  technology  transfer  efforts.  The 
purpose  of  this  section  is  to  detail  the  process  that  will  be  used  to  gather  and  analyze  the 
data  in  support  of  this  research.  This  is  an  exploratory  study  that  will  rely  on  both 
quantitative  and  qualitative  data.  Recent  technology  transfers  conducted  at  Wright 
Laboratory  are  used  as  the  primary  source  of  data.  Data  bases  at  Wright  Laboratory 
provide  secondary  data  on  revenues  attributed  to  given  transfers.  In  addition,  personal 
interviews  with  transfer  focal  points  are  used  to  collect  primary  data  on  other  quantifiable 
and  nonquantifiable  benefits. 

During  this  exploratory  study,  a  ROI  methodology  is  used  to  estimate  the 
expected  returns  on  technology  transfers  within  Wright  Laboratory  (WL).  Recent 
cooperative  research  and  development  agreements  (CRD  As)  are  used  as  a  source  of  data 
for  this  research.  Due  to  time  constraints,  the  opportunities  to  validate  the  ROI  model  are 
limited.  A  potential  follow-on  research  effort  can  validate  estimated  returns  provided  by 
the  methodology  by  examining  the  actual  future  returns  received  at  WL.  Also,  the 
lessons  learned  section  of  the  interview  process  provides  some  additional  research 
avenues. 

Research  Approach 

This  is  an  exploratory  study  that  utilizes  probability  encoding  interview 
techniques  to  gather  data  that  is  not  readily  available  or  located  in  a  database.  Since  the 
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control  by  the  researcher  over  the  variables  is  limited,  it  is  an  ex  post  facto  study.  Once 
data  is  collected  through  the  interview  process  and  probability  distributions  are  postulated 
for  the  estimated  tangible,  quantifiable  benefits,  Monte  Carlo  Simulation  is  used  to 
facilitate  the  ROI  calculations.  This  methodology  is  broken  down  into  three  processes: 
the  identification  of  quantifiable/tangible  and  nonquantifiable/intangible  benefits;  the 
specification  of  those  benefits;  and  the  return  on  investment  calculation. 

Before  a  ROI  methodology  can  be  implemented,  the  tangible  and  intangible 
benefits  must  be  identified  and  quantified.  The  tangible  benefits,  such  as  cash  revenues 
received  in  the  form  of  licensing  fees  and  royalty  payments,  are  available  at  Wright 
Laboratory  for  each  individual  transfer.  Also,  any  obvious  dollar  cost  savings  that  are  to 
be  achieved  as  a  result  of  transfer  activities  are  available  in  the  CRD  A.  The  first  step  in 
the  interview  process  is  the  validation  of  all  benefits  spelled  out  in  the  CRDA.  After  the 
tangible  benefits  and  costs  have  been  identified  and  quantified,  a  structured  interview 
process  is  used  to  identify  other  quantifable/tangible  and  intangible  benefits.  The 
following  list  of  benefits  was  developed  during  an  interview  with  Steve  Guilfoos  AFMC 
TTO/TTR  (15). 

The  list  below  contains  four  main  categories  of  benefits.  The  first  category  is 
revenues  received  from  CRD  As  which  can  be  defined  as  direct  cash  payments  made  to 
WL.  The  second  category,  quantifiable  (non-revenue)  benefits  that  can  be  expressed  as 
dollars,  are  tangible  benefits  that  represent  services  or  products  provided  by  the  outside 
partner  at  no  cost  to  WL  or  payments  made  to  WL  for  the  use  of  its  resources.  The  third 
category,  other  quantifiable  (non-revenue)  benefits,  are  tangible  benefits  that  must  be 


translated  into  dollars  after  being  received.  The  fourth  category,  other  nonquantifiable 
intangible  benefits,  are  real  benefits  that  can  not  be  expressed  in  dollars.  The  list  is  not 
all  inclusive,  but  it  represents  AFMC  TTO  expectations  of  the  anticipated  tangible  and 
intangible  benefits  received  by  the  Air  Force  through  CRD  As. 

Revenues  received  from  CRD  As: 

-  Licensing  Fees 

-  Royalty  Payments 

Quantifiable  (non-revenue/tangible)  benefits  that  can  be  expressed  as  dollars: 

-  Cash  Reimbursables  for  services  provided  by  DoD  personnel  to  the  private 
sector  (ex  consulting  fees) 

-  Work  avoidance  resulting  from  collaborator  efforts  to  be  measured  in  man  hr. 
saved 

-  Cost  avoidance  resulting  from  collaborator  efforts  to  be  measured  in  the  dollar 
value  of  resources  saved  as  a  result  of  the  outside  partner’s  efforts 

Other  quantifiable  (non-revenue/tangible)  benefits: 

-  Productivity  and  efficiency  increases  resulting  from  information  exchange  with 
CRDA  partner 

-  Cost  avoidance  through  more  effective  utilization  of  federal  laboratory 
resources 

-  Time  and  costs  savings  resulting  from  formal  data  exchange  with  CRDA 
partner 

Other  nonquantifiable  (non-revenue/intangible)  benefits: 

-  Improvements  in  managerial  and  business  practices  resulting  from  relationship 
with  CRDA  partner 

-  Improvements  in  the  morale  of  Lab  personnel  through  science  and  technology 
fulfillment 

—  Personnel  see  scientific  goals  being  met 
—  Personnel  see  program  objectives  being  met 
—  Commercialization  of  technologies  developed  in  federal  laboratories 


Interview  Process 

The  interview  process  is  critical  to  building  the  estimates  for  the  intangible 
benefits  of  a  given  transfer.  A  focal  point  or  expert  was  identified  within  Wright 
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Laboratory  for  each  specific  transfer.  Each  focal  point  has  in-depth  knowledge  about 
how  the  transfer  agreements  were  developed  and  what  the  government’s  goals  were 
before  entering  each  agreement.  Once  a  focal  point  is  identified,  the  interview  is 
scheduled. 

The  first  step  of  the  interview  is  a  brainstorming  session  to  identify  the  benefits 
of  the  transfer  that  were  spelled  out  in  the  CRD  A.  Once  the  benefits  are  identified,  they 
are  ranked  or  categorized  based  on  their  importance  to  the  government  and  the  impact 
they  could  potentially  have  on  the  organization.  After  the  potential  impact  has  been 
identified,  the  uncertainty  or  likelihood  of  that  benefit  occurring  must  be  established. 
Probability  encoding  is  used  to  develop  a  distribution  for  the  proposed  benefit. 

“Experts  are  usually  capable  of  making  accurate  forecasts  on  a  future  outcome 
when  the  variable  being  forecast  has  been  defined  in  terms  they  can  understand  (35: 
353).”  The  interview  process  is  divided  into  five  distinct  phases: 

a.  The  Motivating  phase  introduces  the  subject  to  the  encoding  process  by 
explaining  its  purpose  and  its  importance  in  decision  making.  The  motivating 
phase  also  helps  the  interviewer  determine  the  subject’s  bias; 

b.  Structuring  is  the  phase  where  the  benefit  is  defined  and  quantified; 

c.  The  Conditioning  phase  is  used  to  help  the  subject  think  fundamentally  about 
his  judgment  and  to  avoid  cognitive  bias; 

d.  The  Encoding  phase  is  used  to  quantify  the  subject’s  judgment  in 
probabilistic  terms; 

e.  Verifying  is  the  phase  where  the  responses  are  cross  checked  to  ensure  they 
are  valid. 
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The  following  information  describes  the  specific  process  used  to  develop  subjective 
probability  density  functions  for  the  purpose  of  technology  transfer  ROI  estimates. 

Pre-Interview  Research: 

The  first  step  in  the  interview  is  reading  through  the  CRD  A.  This  helps  the  interviewer 
become  knowledgeable  on  what  type  of  technology  is  being  transferred  and  helps  in  the 
formulation  of  some  initial  questions  for  the  respondent.  The  interviewer  reads  through 
the  CRDA  appendix  and  the  CRDA  work  plan  which  contains  a  listing  of  the  expected 
benefits  the  government  was  to  receive  from  the  agreement  and  the  services  the 
government  provided  in  exchange  for  those  expected  benefits. 

Phone  Call  to  POC: 

The  database  listing  provided  by  WL  ORTA  lists  the  POCs  for  each  CRDA.  Once  the 
interviewer  has  read  through  the  CRDA  and  formulated  questions,  the  POC  is  contacted 
and  a  meeting  time  for  the  interview  is  scheduled.  During  the  phone  call,  the  interviewer 
explains  the  purpose  of  the  interview.  The  interviewer  lets  the  respondent  know  that  he 
is  interested  in  verifying  the  benefits  received  by  the  government.  In  addition,  additional 
benefits  that  may  not  have  been  spelled  out  in  the  agreement  are  discussed. 

Interview: 

The  interview  begins  by  discussing  the  CRDA  itself  The  respondent  is  asked  about  the 
technology  involved,  how  the  CRDA  was  formed,  and  the  work  involved  in  putting 
together  the  CRDA.  Questions  are  asked  about  the  outside  partner’s  involvement  in  the 
CRDA  process  (how  interested  were  they  in  the  technology?  did  they  come  to 
government?  etc..).  The  interviewer  then  works  from  the  questionnaire  and  begins  the 
encoding  process. 

The  interviewer  explains  the  data  sheet  and  what  is  expected  in  terms  of  a  probability 
distribution.  The  POCs  usually  have  an  in-depth  understanding  of  statistics  and  have  had 
little  problem  with  quantifying  the  benefits  in  terms  of  a  distribution.  The  following 
listing  describes  the  process: 

a.  Identify  a  starting  point,  usually  a  most  like  likely  estimate  or  end  points 
(extreme  values)  for  the  distribution  provided  by  respondent; 

b.  Develop  an  idea  of  how  the  benefit  or  input  parameter  might  behave 
(positively  or  negatively  skewed,  upper  and  lower  limits,  range  of  values); 

c.  Build  the  estimate  and  sketch  it  out,  beginning  with  worst  case  and  best  case 
scenarios  for  input  parameters; 

d.  Verify  the  estimate  with  the  respondent. 
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After  all  of  the  benefits  have  been  identified  and  quantified,  the  interview  sheet  is 
reviewed  with  the  respondent.  The  interviewer  works  to  get  additional  listings  of  people 
that  have  worked  on  other  CRD  As.  The  final  step  in  the  interview  process  has  been  the 
identification  of  the  lessons  learned  by  the  respondent  during  the  CRDA  process.  The 
goal  of  the  lessons  learned  section  is  to  identify  any  traits  common  to  successful  or 
unsuccessful  CRDAs. 

This  process  leads  to  an  estimated  value  ( a  mean  or  most  likely  value)  and  a  subjective 
probability  density  function  (PDF)  for  each  tangible  and  intangible  benefit  (35:  345). 
These  PDFs  are  used  in  the  simulated  ROI  calculation. 

Once  the  benefits  are  identified  and  specified,  the  potential  dollar  amounts  of  the 
benefits  are  estimated  and  the  probabilities  of  obtaining  these  estimated  outcomes  are  also 
specified.  For  example,  a  triangular  distribution  is  used  where  optimistic,  pessimistic, 
and  most  likely  values  are  identified  for  given  parameter  estimates.  From  this  data,  a 
Monte  Carlo  Simulation  is  used  to  provide  estimated  values  for  given  outcomes  based  on 
the  input  parameters  for  the  specified  distribution.  The  result  of  the  simulation  is  a 
confidence  interval  for  each  outcome  or  parameter  and  an  overall  estimate  of  the  benefits 
received  from  a  CRDA  that  can  be  used  for  decision  making. 

The  ROI  calculation  is  accomplished  once  the  all  of  the  benefits  and  have  been 
identified  and  estimated  and  the  parameter  estimates  and  distributions  have  been 
specified  for  the  quantifiable  benefits.  Microsoft  Excel™  is  used  in  conjimction  with  the 
Crystal  Ball™  program  to  build  spreadsheets  that  perform  the  ROI  calculation. 

Crystal  Ball™  is  a  windows  application  the  was  designed  to  perform  risk  analysis 
in  conjunction  with  Microsoft  Excel™,  thus  extending  the  forecasting  capability  of  a 
spreadsheet.  Two  major  limitations  of  spreadsheets  are  that  they  only  allow  changes  one 
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cell  at  a  time  and  they  do  not  indicate  the  likelihood  or  the  probability  of  achieving  a 
desired  outcome  (10:  8).  Crystal  BalF“  allows  the  analyst  to  describe  a  range  of  possible 
cell  values  in  terms  of  probability  distributions  or  assumptions.  Using  Monte  Carlo 
Simulation,  Crystal  Ball™  shows  the  analyst  the  entire  range  of  possible  outcomes  and 
the  likelihood  of  achieving  them.  This  process  provides  the  analyst  with  a  statistical 
picture  of  the  range  of  possibilities  based  on  the  probability  distributions  associated  with 
each  cell  value  (10:  8-10).  The  benefits,  to  the  decision  maker,  of  using  this  approach  are 
improved  accuracy  and  greater  understanding  of  the  risks  and  returns  associated  with  a 
desired  outcome  (10: 10). 

Data  Requirements 

The  goal  of  this  thesis  is  to  build  a  useful  methodology  that  can  be  employed  by 
the  AFMC  TTO  to  calculate  the  ROI  for  their  transfer  activities.  Monte  Carlo  Simulation 
is  used  in  conjunction  with  discounted  cash  flow  techniques  to  determine  the  ROI  of 
technology  transfer  activities.  In  order  to  build  diis  methodology,  data  on  a  transfer’s 
revenues  and  non-revenue  and  nonquantifiable/intangible  benefits  are  required.  The  data 
on  the  revenues  and  non-revenue  benefits  are  not  consolidated  in  a  database  at  WL. 
Therefore,  this  data  is  obtained  using  the  probability  encoding  interview  techniques 
described  in  Chapter  Two.  Through  the  use  of  the  probability  encoding  techniques,  each 
tangible  benefit  can  be  given  a  quantifiable  value  (usually  a  most  likely  estimate  or  a 
mean).  This  value  is  then  specified  in  terms  of  a  standard  deviation  or  high  and  low 
estimates.  As  a  result  of  the  encoding  process,  each  tangible  benefit  can  be  stated  as  a 
probability  density  function.  This  function  is  then  incorporated  into  the  ROI  calculation. 
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The  result  is  a  ROI  calculation  that  can  be  stated  in  terms  of  probabilities.  The  example 
below  is  a  demonstration  of  aNPV  calculation  using  Monte  Carlo  Simulation. 


The  return  on  investment  calculation  below  was  developed  using  a  500  iteration  Monte 
Carlo  Simulation.  The  intangible  benefits  for  years  1  through  4  are  stated  in  terms  of 
probability  density  functions.  The  shaded  values  shown  in  the  table  below  are  the  mean 
values  for  each  intangible  benefit.  The  total  revenue  and  PV  calculations  were 
calculated  by  summing  the  mean  values  for  each  intangible  benefit.  For  the  example 
below,  there  is  a  50%  probability  of  obtaining  the  NPV  of  $  24,841.11 
Year  1  Year  2  Year  3  Year  4 


Project  Cost  $100,000.00 

Cash  Revenue  $  18,000.00  $12,000.00  $10,000.00 

intangible  Benefit  1  $  12,688.19  $10,716.00  $  7,683.07 
intangible  Benefit  2  $  9,594.75  $  9,545.63  $  8,310.74 

Intangible  Benefit  3  $  5,673.38  $  7,341.47  $  8,387.87 

total  revenue  $  45,956.33  $39,603.09  $34,381.68 


$  8,000.00 
$  7,634;i5 
$  5,870.51 

$10,675.51  PV  of  cash 
flows 

$32,180.17  $124,841.11 


NPV  of  the  $24,841.11 

Project 


FIGURE  3.1  Return  on  Investment  Calculation  Example 

Data  Analysis 

The  most  important  aspect  of  the  methodology  is  the  data  analysis  section. 

The  data  is  analyzed  to  determine  whether  or  not  the  benefits  derived  from  technology 
transfers  are  in  fact  collectable,  quantifiable,  measurable,  and  appropriate.  According  to 
previous  research  by  West  (1995),  there  are  no  formal  data  collection  methods  in  place  at 
WL  to  collect  data  on  the  benefits  received  through  CRD  As.  The  information  provided 
in  the  West  thesis  indicates  that  benefits  or  revenues  received  by  WL  through  technology 
transfers  are  probably  the  best  measure  of  technology  transfer  success.  The  question  that 
must  be  resolved  is;  “Does  the  collection  and  or  estimate  of  revenues  received  by  the 
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government  through  technology  transfers  provide  an  effective  management  decision 
making  tool  that  warrants  the  time  and  effort  used  in  collecting  the  data?” 

During  this  phase  in  the  research,  the  benefits  received  from  transfers  are 
evaluated  using  the  ROI  calculation.  The  transfers  that  show  the  highest  return  on 
investment  are  analyzed  more  closely.  The  focus  of  this  analysis  is  to  determine  if  the 
quantifiable/tangible  benefit  estimates  (input  parameters  or  subjective  PDFs)  are 
reasonable  in  the  first  place.  A  key  point  to  remember  is  that  estimates  developed  using 
subjective  PDFs  are  only  as  good  as  the  parameter  inputs  themselves.  Several  factors 
must  be  considered  to  determine  the  usefulness  of  the  estimates  and  the  subjective  PDFs. 

The  first  factor  that  must  be  considered  is  the  amount  of  and  type  of  bias 
contained  in  the  PDFs.  Biases  and  the  sources  of  biases  must  be  identified  when 
subjective  PDFs  are  used  in  the  ROI  calculations.  As  explained  in  Chapter  Two,  bias  can 
be  prevalent  whenever  individual  expert  opinion  is  being  used  to  develop  PDFs  based 
subjective  assessments  of  potential  outcomes.  The  existence  of  biases  in  an  estimate  will 
often  lead  to  an  imderstatement  of  the  variation  or  cost  risks  associated  with  a  given 
project  (4:  64-67).  Another  factor  that  must  be  addressed  is  whether  or  not  the  type  of 
PDF  used  in  the  estimate  is  appropriate,  applicable,  and  suited  for  cost  risk  analysis.  In 
other  words,  do  the  PDFs  used  to  develop  the  estimate  pass  the  common  sense  test. 

In  addition,  the  variance  contained  in  the  PDFs  should  accurately  represent  the 
respondent’s  confidence  in  the  PDF  inputs.  The  researcher  should  expect  the  amount  of 
variance  contained  in  the  PDFs  to  coincide  with  the  amoimt  of  imcertainty  in  the 
estimate.  Higher  degrees  of  imcertainty  should  be  reflected  by  higher  variances  in  the 
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subjective  PDFs  and  wider  confidence  intervals  for  the  ROI  estimate.  In  cases  where 
there  is  a  high  degree  of  xmcertainty  in  the  quantifiable  benefit  parameter  estimates  and 
the  ROI  calculation  is  favorable,  revisiting  the  estimate  to  identify  any  parameter 
estimates  that  are  overly  optimistic  and  contain  high  degrees  of  variation  is  advisable. 
This  is  simply  meant  to  be  a  system  of  checks  and  balances  to  guard  against  overly 
optimistic  estimates  that  could  potentially  lead  to  poor  decisions  by  managers. 

Another  factor  to  be  studied  is  any  estimate  that  contained  a  high  degree  of 
correlation  between  parameter  inputs.  A  certain  transfer  may  provide  a  favorable  ROI 
number,  but  it  may  contain  high  degrees  of  correlation  between  input  parameters.  The 
existence  of  correlation  between  parameters  could  potentially  lead  to  a  higher  than 
expected  amounts  of  variation  in  the  ROI  calculation.  Estimates  that  contain  high 
degrees  of  variation  resulting  from  correlation  between  parameter  inputs  may  not  be 
reliable  for  the  purposes  of  managerial  decision  making  and  should  be  examined  more 
closely. 

Expectations 

The  interview  process  should  provide  reliable  and  effective  PDFs  that  can  be  used 
in  the  ROI  calculation.  The  shape  of  the  distributions  should  be  defined  based  on  the 
logic  of  the  given  quantifiable/tangible  benefit  and  the  subject’s  expert  opinion  of 
potential  outcomes.  The  ROI  calculations  should  demonstrate  that  the  technology 
transfer  activities  taking  place  within  AFMC  are  beneficial  to  the  DoD,  the  Air  Force  and 
the  military  laboratories. 
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Limitations 

Using  probability  encoding  techniques  to  develop  subjective  PDFs,  and  then 
using  these  PDFs  in  a  ROI  calculation  that  incorporates  Monte  Carlo  Simulation  can  be  a 
difficult  task.  The  ROI  calculation  is  only  as  reliable  as  the  subjective  PDFs  that  go  into 
it.  This  is  an  advanced  approach  in  the  area  of  risk  analysis  and  there  are  several  reasons 
why  this  approach  is  not  often  used.  First,  specification  of  a  subjective  PDF  is  not  an 
easy  task;  much  of  the  burden  lies  Avith  the  interviewer.  An  additional  factor  that  must  be 
considered  is  that  variables  are  often  correlated  to  one  another.  Unfortunately  these 
correlations  may  be  difficult  to  identify  in  the  first  place  and  once  identified,  specification 
errors  are  common.  Finally,  the  use  of  simulation  may  not  lead  to  a  clear  cut  decision 
rule  for  decision  makers.  A  mean  value  for  the  ROI  calculation  is  obtained  and  the 
probabilities  associated  Avith  certain  outcomes  are  identified.  Unfortunately,  this 
information  may  not  be  adequate  for  some  senior  decision  makers. 

Summary 

This  chapter  presented  the  research  approach,  sample  population,  interview 
process,  data  requirements,  data  analysis,  expectations  and  limitations.  The  method  of 
collecting  data  is  focused  interviews  Avith  knowledgeable  points  of  contact.  Probability 
encoding  is  used  to  define  the  assumptions  associated  Avith  each  quantifiable  outcome. 
The  data  is  analyzed  through  the  construction  of  a  ROI  model  that  can  be  utilized  by 
AFMC  to  identify  the  technology  transfer  benefits  received  by  the  Air  Force  from 
CRD  As.  Chapter  four  presents  the  results  of  the  data  collection  and  analysis  effort. 
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rV.  Results 


Introduction 


The  objective  of  this  research  is  to  collect  data  on  the  benefits  received  by  the  Air 
Force  through  the  use  of  cooperative  research  and  development  agreements  (CRD  As) 
between  one  government  laboratory,  Wright  Laboratory  (WL),  Wright-Patterson  Air 
Force  Base,  Ohio,  and  its  collaborators  in  the  private  sector.  The  data  collected  will 
provide  insight  into  the  success  or  failure  of  the  technology  transfer  program  within  WL. 
Research  objectives  are  as  follows: 

1 .  Identify  the  benefits  that  were  expected  to  be  received  by  the  Air  Force  or  WL 
jfrom  involvement  in  technology  transfer  with  outside  partners. 

2.  Determine  whether  or  not  the  expected  benefits  spelled  out  in  the  CRD  As  were 
actually  realized  during  the  technology  transfer  process  in  WL. 

3.  Quantify  the  benefits  received  by  WL  and  utilize  the  return  on  investment 
methodology  described  in  the  previous  chapter  to  provide  measures  that  can  be  used  to 
assess  the  success  of  our  government’s  technology  transfer  efforts. 

4.  Analyze  the  data  provided  in  the  interviews  to  determine  whether  or  not  the 
benefits  derived  fi-om  technology  transfers  are  in  fact  collectable,  quantifiable, 
measurable,  and  appropriate. 

5.  Identify  any  lessons  learned  by  WL  personnel  about  the  CRD  A  process  or  specific 
reasons  why  a  given  CRDA  was  a  success  or  failure. 


This  chapter  discusses  the  data  collected  during  15  personal  interviews  with 
CRDA  POCs  in  WL.  This  exploratory  study  is  discussed  through  three  main  areas.  First, 
the  method  of  collecting,  analyzing,  and  presenting  the  data  is  provided.  Second,  the  data 
is  discussed  with  respect  to  three  categories  of  benefits  received  by  WL  as  a  result  of  the 
technology  transfer  process.  These  three  categories  are: 

1 .  CRD  As  that  resulted  in  no  benefit  to  WL; 

2.  CRDAs  that  resulted  in  or  are  expected  to  result  in  nonquantifiable  benefits  to 
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WL; 

3.  CRD  As  that  have  resulted  in  or  are  expected  to  result  in  revenues  or 
quantifiable  benefits  to  WL. 

Finally,  the  ROI  calculations  'will  be  revisited  and  analyzed  to  address  any  areas 
of  concern  like  bias  'within  the  subjective  PDFs,  correlation  between  assumptions, 
variance  in  the  estimate,  and  reasonableness  of  the  ROI  calculation.  The  three  categories 
and  the  benefits  received  are  analyzed  in  accordance  with  the  lessons  learned  section  of 
the  interview  to  determine  if  there  are  traits  common  to  transfers  that  provide  higher  than 
normal  returns  to  the  government. 

Data 

The  data  used  in  this  study  comes  from  the  15  interviews  conducted  with  CRD  A 
points  of  contact  (POCs).  The  data  is  self-reported  and  internally  consistent  with  the 
interview  format  developed  and  explained  in  the  previous  chapter.  No  additional 
procedures  or  processes  were  used  to  verify  the  data  reported  by  WL  POCs  dxrring  the 
interviews.  All  of  the  respondents  were  frank,  honest,  and  candid  about  the 
development,  benefits,  and  administration  of  the  CRD  As  they  managed.  They  were  all 
extremely  cooperative  in  sharing  their  experiences  and  insights  about  the  CRDA  process 
and  the  relationship  between  Air  Force  and  the  outside  partner  or  collaborator.  The 
cooperation  and  honesty  of  the  respondents  leads  this  researcher  to  believe  that  the 
information  and  responses  provided  during  the  interviews  are  true  to  the  best  of  the 
respondents’  knowledge. 

The  summary  of  the  CRDA  POC  interview  data  is  presented  in  Appendix  B.  The 
data  was  collected  using  the  CRDA  interview  form  presented  in  Appendix  A  and  the 
probability  encoding  process  described  in  Chapter  Three. 
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Benefits  to  WL 


The  results  of  the  data  collection  process  showed  that  WL  is  in  many  cases 

actively  and  effectively  transferring  technology  to  the  private  sector.  Of  the  15  CRDAs 

that  were  studied  by  this  researcher,  only  four  had  resulted  in  no  benefits  to  the 

government  or  WL.  Of  the  remaining  1 1  CRDAs,  nine  had  either  already  resulted  in 

revenues  to  the  government  or  the  respondent  felt  that  it  was  appropriate  to  calculate 

future  revenue  estimates  resulting  from  the  CRDA  using  the  ROI  methodology  explained 

in  Chapter  Three.  See  graph  and  table  below: 

No  Benefits  to  Govt  4 

Nonquantifiable  Benefits  2 
Quantifiable  and  Non-  9 

Quantifiable  Benefits 


%  of  CRDAs  in  each  of  the  3  categories 

No  Benefits  to 
Govt. 

27% 

Quantifiable  and 
Nonquantifiable 
Benefits 
60% 


Benefits  Only 
13% 


FIGURE  4.1  Number  and  Percentage  of  CRDAs  in  each  of  the  Three  Categories 
The  percentages  shown  in  the  above  graph  are  not  necessarily  representative  of  eill 
CRDAs  developed  and  administered  at  WL.  More  extensive  sampling  would  be  required 
to  determine  whether  or  not  the  sample  shown  above  is  representative  of  all  CRDAs 
developed  at  WL. 
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The  next  step  in  analyzing  the  data  is  to  identify  the  benefits  that  are  being 
received  by  the  government  or  WL.  By  analyzing  the  1 1  CRD  As  that  were  providing 
benefits  to  WL,  a  listing  of  benefits  could  be  identified.  The  benefits  are  categorized 
according  to  the  listing  of  expected  benefits  presented  in  Chapter  Three.  Additional 
benefits  not  specified  in  Chapter  Three  have  been  added  to  this  listing.  The  interview 
summaries  contained  in  Appendix  B  provide  comprehensive  explanations  of  the  benefits 
received  by  WL  as  a  result  of  the  15  CRD  As  used  in  this  study.  The  categorized  listing 
below  is  followed  by  explanations  and  examples  of  the  benefits: 

The  benefits  contained  in  the  following  table  are  a  summation  of  all  benefits  received  and 
are  not  traced  specifically  to  individual  CRDAs.  In  other  words,  there  were  some 
CRDAs  that  provided  multiple  benefits  in  the  same  specific  category  (e.g,  resource 
savings).  Those  benefits  were  accounted  for  each  time  they  occurred  and  were  not 
consolidated  when  they  resulted  from  only  one  CRDA. 

TABLE  4. 1  Benefits  Received  by  the  Air  Force/WL 

Benefits  Received  by  the  Air  Force/WL 

Revenues 

-  Royalty  Payments 
Quantifiable  Benefits  in  $ 

-  Cash  Reimbursements 

-  Work  Savings 

-  Resource  Savings 
Other  Quantifiable  Benefits 

-  Productivity  Increases 

-  Effective  Resource  Utilization 

-  Data  Exchange  savings 
Nonquantifiable  Benefits 

-  Public  Relations  and  Goodwill 

-  Enhanced  Morale,  Knowledge,  or 
Expertise  of  Lab  Personnel 

-  Dissemination  of  Technology 

-  Improvements  in  Technology 

-  Use  of  Additional  Technologies 

-  Enhanced  Society  Wealth 

-  Follow-on  CRDA  Activities 

-  Transfer  Process  Improved 


4 

4 

14 

2 

4 

8 

11 

1 

7 

3 

32 

6 

6 

4 
6 
1 
1 
4 
4 
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The  revenues  category  is  made  up  of  expected  royalty  payments  to  be  received  by 
WL  as  a  result  of  transfer  activities.  Four  of  the  1 1  respondents  responsible  for  benefit- 
producing  CRD  As  felt  that  the  potential  possibility  of  royalty  revenues  was  high  enough 
to  warrant  the  inclusion  of  a  royalty  revenue  stream  in  the  ROI  calculation.  This  data 
does  not  necessarily  represent  revenues  that  have  been  received  by  WL  in  the  form  of 
royalty  payments.  This  data  only  represents  the  fact  that  there  is  the  potential  in  a  given 
CRD  A  for  a  revenue  producing  royalty  stream. 

The  second  category,  quantifiable  benefits  that  can  be  expressed  in  dollars, 
contains  cash  reimbursables,  work  savings  resulting  from  collaborator  efforts,  and 
resource  savings  resulting  from  collaborator  efforts.  Examples  of  cash  reimbursables 
include  consulting  fees  or  payments  made  by  the  outside  partners  for  WL 
expertise/resources.  Examples  of  work  savings  include  man-years  of  effort  (usually 
quantified  in  terms  of  a  salaried  employee)  saved  in  the  marketing  or  enhancement  of  a 
technology  or  the  development  of  a  database  by  the  collaborator.  Examples  of  resource 
savings  include  work  done  by  the  outside  partner  to  improve  a  technology,  making  it 
more  operationally-effective  or  cost-effective  for  government  use.  Other  examples 
include  manuals,  testing,  or  other  resources  provided  to  WL  at  no  cost  as  part  of  the 
CRDA. 

The  third  category  of  benefits,  other  quantifiable  benefits,  includes  productivity 
increases,  more  effective  utilization  of  government  resources,  and  time  and  costs  savings 
resulting  from  formal  knowledge  and  data  exchange  with  CRDA  partners.  Examples  of 
cost  savings  due  to  effective  utilization  of  WL  resources  include  additional  use  of  testing 
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facilities  at  WL  resulting  from  CRD  As  and  the  use  of  equipment  already  purchased  by 
WL  that  was  utilized  more  effectively  as  a  result  of  the  CRDA.  Examples  of  time  and 
cost  savings  resulting  from  data  exchange  and  knowledge  transfer  with  outside  partner 
include  insights  gained  about  commercial  aircraft  industry. 

The  fourth  category  of  benefits,  other  nonquantifiable  benefits,  are  made  up  of 
benefits  that  are  real  and  valuable  to  the  government  and  WL.  The  benefits  include  but 
are  not  limited  to  the  above  listing.  They  are  all  self  explanatory.  However  the  three 
most  common  nonquantifiable  benefits:  public  relations  and  goodwill,  enhanced  morale, 
expertise,  and  knowledge  of  WL  personnel,  and  technology  enhancements  seemed 
extremely  important  to  most  of  the  respondents.  At  least  one  of  these  three  benefits  were 
identified  in  each  of  the  1 1  benefit-producing  CRD  As.  Many  of  the  respondents  pointed 
out  that  the  fact  that  they  were  actively  and  effectively  doing  technology  transfer  was 
important  to  the  WL  mission  and  society  as  a  whole.  On  several  occasions,  respondents 
mentioned  that  these  benefits  may  even  be  more  important  to  WL  than  the  quantifiable 
categories  of  benefits  mentioned  above. 

After  compiling  the  data,  it  is  clear  that  the  nonquantifiable  benefits  were  more 
common  than  the  other  categories  of  benefits.  The  graph  on  the  following  page  shows 
the  overall  number  of  benefits  that  were  identified  in  each  of  the  four  main  categories. 
The  research  indicates  that  there  are  real  benefits  to  WL  resulting  from  technology 
transfer.  The  graph  shows  that  these  benefits  are  in  many  cases  nonquantifiable.  In  fact, 
the  nonquantifiable  benefits  outnumbered  the  other  three  categories  combined.  Because 
over  50%  of  the  benefits  identified  are  nonquantifiable,  the  use  of  a  basic  ROI 
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methodology  may,  at  times,  be  inappropriate  or  difficult  to  apply  when  used  exclusively 

to  evaluate  the  benefits  received  by  the  Air  Force  through  technology  transfer. 

This  graph  represents  the  summation  of  all  benefits  received  by  WL  through  CRDA 
activities.  Benefits  are  accumulated  in  total  and  not  traced  to  individual  CRDAs. 
Therefore,  several  of  the  CRDAs  provided  multiple  benefits  by  major  category. 


Major  Benefit  Categories 


Benefits 


FIGURE  4.2  Major  Categories  of  Benefits  Received  Through  Technology  Transfer 
When  looking  at  the  major  categories  of  benefits  received  by  WL  through 
technology  transfer,  it  becomes  clear  that  the  intangible,  non-quantifiable  benefits  are 
extremely  common.  The  respondents  explained  that  these  non-quantifiable  benefits  are 
important  to  WL  and  the  WL  mission.  The  next  step  in  analyzing  the  benefits  was  to 
identify  the  specific  benefits  that  were  most  commonly  received  by  WL  through 
technology  transfer.  The  research  indicates  that  the  most  common  benefit  received  by 
WL  through  technology  transfer  is  the  resource  savings  experienced  as  a  result  of 
collaboration  with  outside  partners.  The  resources  saved  through  technology  transfer  can 
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be  quantified  in  many  cases  and  represent  hard  evidence  that  the  Air  Force  is  getting 

some  return  on  their  technology  transfer  efforts.  The  graph  below  represents  the  most 

common  specific  benefits  received  by  WL  through  the  1 1  benefit  producing  CRD  As. 

The  benefits  contained  in  the  graph  below  represent  a  summation  of  all  benefits  received 
in  specific  categories  and  are  not  traced  specifically  to  individual  CRDAs.  In  other 
words,  there  were  some  CRDAs  that  provided  multiple  benefits  in  the  same  specific 
category  (i.  e.  resource  savings).  Those  benefits  were  accounted  for  each  time  they 
occurred  and  were  not  consolidated  when  they  resulted  from  only  one  CRDA. 


Most  Common  Benefits  to  WL 


#  of  Benefits 


■  -  Transfer  Process  Improved 

■  -  Foliow-on  CRDA  Activities 

0  -  Improvements  in  Technology 

■  -  Dissemination  of  Technology 

0  -  Morale/expertise  of  Lab 
Personnel 

■  -  Public  Relations  and  Goodwill 

□  -  Effective  Resource  Utilization 

□  -  Resource  Savings 

■  -Work  Savings 

B  -  Royalty  Payments 


FIGURE  4.3  Most  Common  Benefits  Received  by  WL  Through  Technology  Transfer 
The  fact  that  there  are  quantifiable  benefits  being  received  by  WL  through  their 
technology  transfer  efforts  provides  justification  for  the  use  of  the  ROI  methodology 
described  in  Chapter  Three.  Effective  resource  utilization,  resource  savings,  work 
savings,  and  royalty  payments  are  all  factors  that  federal  laboratory  decision  makers  can 
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look  at  more  closely  as  future  decision  making  criteria.  Additional  revenues  to  WL  and 
costs  savings  resulting  from  technology  transfer  efforts  will  allow  WL  to  ration  out  their 
R&D  dollars  more  effectively.  In  many  cases,  WL  has  been  able  to  utilize  CRDA 
revenues  to  fund  capital  expenditures  that  would  have  not  been  possible  due  to  funding 
limitations. 

CRDA  Revenues 

In  many  cases,  the  expected  benefits  spelled  out  in  the  CRDA  work  plans  were 
realized  or  are  expected  to  be  realized  by  the  government.  The  following  table  displays 
the  estimated  mean  net  present  value  of  the  quantifiable  benefits  received  in  the  9  revenue 
producing  CRD  As.  The  values  range  from  a  low  of  $  2,000  to  a  high  of  $  834,574.  The 
estimates  were  developed  using  the  ROI  methodology  described  in  Chapter  Three. 
Appendix  B  contains  the  spreadsheet  calculations  and  Crystal  Ball  reports  for  each 
estimate.  In  many  cases,  the  cash  flow  estimates  were  developed  using  the  probability 
encoding  techniques  described  by  Spetzler  (35;  350).  Subjective  PDFs  were  built  based 
on  the  expert  opinion  of  the  CRDA  POCs  (35:  350).  There  were  no  real  difficulties  in 
obtaining  the  estimates  because  all  of  the  respondents  had  some  knowledge  of  statistics 
and  probability  density  functions.  In  order  to  guard  against  overly  optimistic  estimated 
values,  subjective  PDFs  were  built  only  for  potential  benefits  in  cases  where  the 
respondents  felt  there  was  a  strong  chance  of  the  benefits  being  realized.  The  values 
listed  below  are  an  accurate  and  documented  estimate  of  the  revenues  expected  from  the 
individual  CRDAs. 

The  revenues  already  realized  represent  actual  dollars  or  estimated  actual  dollars  that 
have  already  been  received  by  WL.  In  many  cases,  these  revenues  were  in  the  form  of 
payments  to  WL  for  services,  resources,  or  expertise  provided  under  the  terms  of  the 
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CRDA.  in  other  cases,  these  revenues  already  realized  were  calculated  using  probability 
encoding  techniques  and  Monte  Carlo  Simulation.  The  estimated  revenues  represent 
revenues  contained  in  the  ROI  calculation  that  are  expected  in  the  future  but  have  not  yet 
been  realized. 

TABLE  4.2  Estimated  Net  Present  Value  of  Revenues  Received  Through  CRD  As 


Revenues  Already 


CRDA# 

Realized 

94 

-173- 

WL 

$ 

45,000.00 

93 

-335 

WL 

$ 

44,094.00 

95 

-004 

WL 

$ 

2,000.00 

94 

-132 

WL 

$ 

43,551.00 

95 

-335 

WL 

$ 

21,000.00 

95 

-075 

WL 

$ 

- 

95 

-201 

WL 

$ 

- 

94 

-083 

AS 

$ 

96,400.00 

96 

-000 

WL 

$ 

- 

Estimated  Mean  Net  Present 
Revenues  Value  Estimate 


$ 

$ 

45,000.00 

$  36,151.00 

$ 

80,245.00 

$ 

$ 

2,000.00 

$ 

$ 

43,551.00 

$  95,123.00 

$ 

116,123.00 

$  249,938.00 

$ 

249,938.00 

$  834,574.00 

$ 

834,574.00 

$  119,228.00 

$ 

215,628.00 

$  246,826.00 

$ 

246,826.00 

r 


There  is  a  wide  disparity  between  the  revenues  expected  from  the  individual 

CRD  As.  The  estimated  royalty  revenue  streams  are  the  main  drivers  in  the  high  end 

estimates.  There  is  a  greater  degree  of  uncertainty  involved  with  these  estimates  because 

the  cash  flow  estimates  are  based  upon  unknown  future  demand  for  the  goods  and 

services  of  the  outside  partner.  The  four  CRD  As  with  estimated  royalty  revenue  streams 

are  all  over  $  200,000  in  mean  estimated  net  present  value.  The  graph  below  displays  and 

emphasizes  the  disparity  between  the  expected  benefits  of  the  five  none  royalty  revenue 

stream  CRD  As  and  the  expected  benefits  of  the  four  royalty  revenue  stream  CRD  As. 

The  four  CRD  As  containing  estimates  of  royalty  revenue  streams  are: 

CRDA  #94 -083 -AS 
CRDA  #  96  -  000  -  WL 
CRDA  #  95  -  075  -  WL 
CRDA  #  95  -  201  -  WL. 
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CRDA  Benefits 
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FIGURE  4.4  Estimated  Mean  Net  Present  Values  for  Revenue  Producing  CRD  As 

With  the  exception  of  CRDA  95  -  004  -  WL,  the  estimated  mean  net  present  value 
estimates  are  substantial  in  terms  of  dollar  value  provided  to  WL  and  the  Air  Force.  Due 
to  the  high  dollar  values  contained  the  ROI  calculations,  further  evaluation  of  the 
estimates  was  necessary.  In  order  to  determine  the  usefulness  of  the  ROI  methodology  as 
a  management  decision  making  tool,  the  data  was  analyzed  based  on  the  following 
criteria: 

a.  Bias  within  the  estimates; 

b.  Specification  of  the  PDFs—are  the  PDFs  reasonable  and  appropriate? 

c.  Variance  in  the  estimate-does  it  coincide  with  estimate  uncertainty? 

d.  Correlation  between  input  parameters. 
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The  data  summaries  and  Crystal  Ball  reports  (see  Appendix  B)  were  utilized  to 
conduct  data  analysis. 

Estimate  Bias 

Any  methodology  that  uses  the  subjective  inputs  of  an  expert  to  develop  estimates 
of  an  outcome  based  on  the  probability  of  future  events  is  subject  to  biases.  Research  by 
Biery  and  Spetzler  has  shown  that  experts  consistently  display  bias  when  estimating  the 
probability  of  future  events.  They  explain  that  these  biases  are  usually  not  due  to  an 
expert’s  failure  to  remain  objective,  but  are  part  of  a  systematic  cognitive  process  (4:  64; 
35:  350).  Much  of  the  responsibility  for  removing  bias  from  a  subjective  input  lies  with 
the  researcher  conducting  the  interview. 

According  to  research  conducted  by  Spetzler  and  Biery  three  of  the  most  common 
biases  exhibited  in  cost  analysis  are  representativeness  where  the  researcher  and 
respondent  become  overreliant  on  certain  information  and  neglect  other  information, 
availability  where  the  most  recent  events  are  recalled  in  order  to  build  distributions,  and 
anchoring  where  a  point  estimate  is  first  determined  and  a  distribution  is  then  built 
aroxmd  the  point  estimate  process  (4:  64;  35:  350). 

The  most  difficult  biases  to  overcome  in  building  PDFs  for  the  ROI  calculations 
were  representativeness  and  anchoring.  In  most  cases,  availability  bias  was  not  a  factor 
because  the  respondents  were  providing  data  on  situations  that  they  had  not  dealt  with  in 
the  past.  Therefore,  there  were  no  recent  events  to  be  recalled  when  building  the 
distributions.  Anchoring  bias  was  minimized  in  most  cases  by  discussing  the  end  points 
of  the  distributions  prior  to  the  most  likely  values  or  mean  values  of  the  distributions. 
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Due  to  the  inexperience  of  the  researcher,  there  were  several  occasions  where  the 
distributions  were  built  by  first  identifying  the  most  likely  values  and  then  the  end  points. 
When  this  occurred,  the  distributions  were  often  expanded  to  provide  greater  variances 
and  extend  the  range  of  the  end  points  of  the  distribution  provided  by  the  respondent. 

This  allowed  for  a  greater  variance  in  the  estimate  and  minimized  the  possibility  of 
understating  the  entire  range  of  possible  outcomes.  For  example,  on  more  than  one 
occasion,  a  work  savings  was  quantified  in  terms  of  a  salaried  employee.  When  the 
respondent  provided  an  estimated  man-year  of  effort  in  terms  of  a  most  likely  salary  and 
pessimistic  and  optimistic  end  points,  the  end  points  were  further  extended  to  include 
salaries  considered  to  be  out  of  the  norm.  On  most  occasions,  the  respondents  felt  that 
the  revised  distribution  was  acceptable.  Because  CRDAs  are  new  and  different  work 
situations  for  most  of  the  respondents,  there  is  a  possibility  of  some  representativeness 
bias  in  the  ROI  calculations.  The  methods  described  above  were  also  used  in  order  to 
compensate  for  the  possibility  of  representativeness  bias. 

PDF  Specification 

Only  two  types  of  distributions  were  used  in  building  the  ROI  calculations.  On 
several  occasions,  the  normal  distribution  was  used.  The  normal  distribution  was  used  in 
situations  where  the  respondent  felt  there  was  an  equal  likelihood  of  the  estimated  value 
falling  above  or  below  the  mean  and  a  standard  deviation  could  be  estimated. 

The  triangular  distribution  was  the  most  widely  used  distribution  in  this  study. 
According  to  Biery  (1992),  the  triangular  distribution  has  several  advantages  over  other 
distributions  (4:  65).  For  instance,  it  is  easy  to  manipulate  mathematically  and  does  not 
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require  the  analyst  to  provide  additional  information  like  parameter  shape,  and  unlike  the 
normal  distribution,  it  allows  for  skewdness  in  the  parameter  input. 

The  respondents  were  all  extremely  helpful  in  developing  the  distributions.  The 
distributions  were  specified  in  terms  that  the  respondents  could  easily  understand.  Most 
inputs  were  put  in  terms  of  units  sold,  dollars  paid  for  resources,  or  time  spent  in  terms  of 
hours  or  years  of  effort.  All  of  the  input  parameters  were  specified  and  then  revisited  to 
assess  reasonableness.  In  every  case,  the  respondents  felt  that  the  input  parameters 
provided  a  fair  and  reasonable  range  of  possible  outcomes. 

Estimate  Variance 

The  variances  contained  in  the  ROI  calculations  were  based  upon  the  uncertainty 
of  the  input  parameters.  There  were  two  basic  categories  of  input  parameters:  the 
speculative  and  the  well-defined.  Examples  of  the  speculative  inputs  include  estimates  of 
future  demand  for  the  products  or  services  resulting  from  the  privatization  of  the 
technology.  Examples  of  well-defined  input  parameters  include  payments  to  WL  for 
testing  that  has  already  been  scheduled  as  part  of  the  CRDA  or  work  savings  quantified  in 
terms  of  a  salaried  employee,  resulting  from  the  collaborator’s  effort  during  the  CRDA. 

The  distinction  between  the  two  types  of  inputs  is  easily  made  because  the  well- 
defined  inputs  are  based  on  events  that  have  or  are  certain  to  occur.  These  events  are  also 
expressed  in  familiar  terms  by  the  respondents.  The  respondents  have  well  defined 
knowledge  on  the  cost  of  tests  conducted  at  WL  or  the  salaries  paid  to  engineers  or 
researchers  at  WL  for  work  performed.  Speculative  inputs  are  based  on  relatively 
unknown  future  events  that  may  or  may  not  occur.  Of  the  nine  revenue  producing 
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CRD  As,  only  four  contained  speculative  inputs.  These  speculative  inputs  were  limited  to 
the  units  sold  category  of  input  parameters  which  were  used  to  calculate  royalty  revenues. 
Therefore,  the  four  CRD  As  that  contained  royalty  revenues  in  the  ROI  calculation 
contained  the  highest  degrees  of  variance  in  the  estimate.  It  is  important  to  note  that  the 
four  CRD  As  expected  to  provide  royalty  revenues  also  have  provided  or  are  expected  to 
provide  additional  more  well  defined  benefits  in  the  form  of  work  or  resource  savings. 
This  being  the  case,  WL  is  almost  certain  to  receive  revenues  in  some  form  from  the  four 
CRD  As  containing  the  highest  amounts  of  variance  in  the  estimated  ROI  calculation. 

The  table  below  consolidates  the  most  relevant  information  contained  in  the  Crystal  Ball 
reports  for  each  individual  CRD  A. 


TABLE  4.3  Consolidation  of  Crystal  Ball  Report  Information 


Mean  Net  Present  Range  Range  Range  Mean  Std.  Variance 


CRDA# 

Value  Estimate 

Min 

Max 

Width 

Error 

in  Millions 

94-173-WL 

$ 

45,000.00 

$ 

45,000 

$ 

45,000 

$ 

- 

$ 

- 

93-335-WL 

$ 

80,245.00 

$ 

68,998 

$ 

98,624 

$ 

29,626 

$ 

132 

$ 

17 

95-004-WL 

$ 

2,000.00 

$ 

2,000 

$ 

2,000 

$ 

- 

$ 

- 

94-132-WL 

$ 

43,551.00 

$ 

28,775 

$ 

55,627 

$ 

26,852 

$ 

182 

$ 

33 

95-335-WL 

$ 

116,123.00 

$ 

104,327 

$ 

131,061 

$ 

26,734 

$ 

187 

$ 

35 

95-075-WL 

$ 

249,938.00 

$ 

154,870 

$ 

342,666 

$ 

187,796 

$ 

1,120 

$ 

1,254 

95-201  -  WL 

$ 

834,574.00 

$ 

523,003 

$ 

1,076,825 

$ 

553,822 

$ 

3,236 

$ 

10,476 

94 -083 -AS 

$ 

215,628.00 

$ 

171,641 

$ 

261,574 

$ 

89,933 

$ 

458 

$ 

210 

95-000-WL 

$ 

246,826.00 

$ 

192,733 

$ 

293,790 

$ 

101,057 

$ 

544 

$ 

296 

On  the  surface,  the  amount  of  variance  in  the  ROI  calculations  appears  to  be 


appropriate  given  the  uncertainty  of  the  input  parameters  and  the  magnitude  of  the 


estimated  values  of  the  ROI  calculations.  The  above  table  displays  the  fact  that  the  range 


width,  mean  standard  error,  and  the  estimate  variances  are  of  greater  magnitude  for  the 
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four  CRD  As  containing  inputs  on  royalty  revenues.  These  are  as  expected  given  the 
amount  of  uncertainty  involved  with  estimating  units  sold  in  the  future. 

Input  Parameter  Correlation 

Correlation  can  be  defined  as  the  systematic  relationship  or  level  of 
interdependence  existing  between  to  variables  or  input  parameters.  The  existence  of 
correlation  between  input  parameters  will  increase  the  magnitude  of  the  total  variance  of 
an  estimate.  In  cases  where  it  is  assumed  that  all  of  the  input  parameters  are  statistically 
independent  from  one  another,  the  total  estimate  variance  will  be  understated  if 
correlation  does,  in  fact,  exist  between  input  parameters. 

It  is  believed  that  there  is  at  least  some  correlation  in  the  parameter  estimates  for 
units  sold  by  the  outside  partner.  In  other  words,  it  is  probably  unrealistic  to  expect  that 
the  demand  for  goods  and  services  produced  by  the  outside  partner  would  be  statistically 
independent  from  year  to  year.  In  computing  the  ROI  for  CRD  As  containing  estimates  of 
royalty  revenues,  at  least  some  correlation  was  assumed  between  the  imits  sold  input 
parameters  from  year  to  year.  The  estimated  correlations  between  input  parameters  are 
shown  in  the  Crystal  Ball  reports  provided  in  Appendix  B.  The  correlation  values 
between  input  parameters  range  from  .15  to  .35,  indicating  weak  to  slightly  moderate 
correlation  between  input  parameters. 

As  expected,  the  magnitude  of  the  variances  did  increase  in  the  ROI  estimates 
where  correlation  between  input  parameters  was  assumed.  However,  it  must  be  noted 
that  the  magnitude  of  the  variances  contained  in  the  ROI  estimates  was  already  expected 
due  to  the  degree  of  imcertainty  involved  in  estimating  the  input  parameters  for  units 
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sold.  Therefore,  the  correlation  contained  in  these  estimates  was  not  the  sole  contributing 
factor  towards  the  disparity  between  variances  in  the  estimates.  The  graph  below 
provides  a  visual  representation  of  the  disparity  between  the  variances  contained  in  the 
ROI  estimates  for  each  revenue  producing  CRD  A.  The  variances  below  are  presented  in 
millions  of  dollars. 

CRDA  #95  -  201  -  WL  contained  a  variance  of  $10,476,257,251  and  has  been  left  out 
from  the  graph  below.  The  magnitude  of  this  data  point  made  a  graphical  display  of  the 
CRDA  variances  ineffective  because  none  of  the  other  variances  were  of  the  same  scale 
on  the  graph.  The  three  largest  variances  in  the  graph  below  are  attributed  to  the  other 
three  CRDAs  that  contain  estimated  royalty  revenue  streams. 


Variance  in  CRDA  Estimates 


FIGURE  4.5  Variance  in  Millions  of  Dollars  for  CRDA  ROI  Estimates 


Several  conclusions  can  be  drawn  from  reviewing  the  ROI  estimates.  First  of  all, 
there  is  a  wide  range  of  estimated  revenues  from  the  CRD  As  in  this  sample.  Second,  the 
estimated  dollar  values  of  the  high  revenue  producing  CRD  As  are  subject  to  a  great  deal 
of  speculation  and  variance.  However,  each  of  these  CRD  As  do  contain  well  defined 
input  parameters  that  represent  significant  work  or  resource  savings  to  WL.  Finally,  the 
respondents  were  successful  in  specifying  PDFs  that  were  applicable,  appropriate,  and 
usable  for  the  ROI  methodology  described  in  Chapter  Three. 

Lessons  Learned 

Each  interview  session  concluded  with  the  respondent  providing  lessons  learned 
from  the  development  and  administration  of  his/her  individual  CRD  As.  For  reporting 
purposes,  the  lessons  learned  will  be  categorized  according  to  the  success  of  CRD  As. 

The  lessons  learned  from  the  unsuccessful  CRD  As  will  focus  on  reasons  why  the  CRDA 
failed  to  provide  benefits  to  WL.  The  lessons  learned  from  the  successful  CRD  As  will 
focus  on  reasons  why  the  relationship  with  the  outside  partner  worked  to  benefit  WL. 

The  common  theme  to  all  of  the  unsuccessful  CRD  As  was  that  the  outside  partner 
appeared  to  be  motivated  solely  by  profits  and  displayed  no  real  interest  in  helping  WL  to 
further  develop  the  technology.  In  one  case,  the  outside  partner  simply  did  not  have  the 
resources  to  work  on  the  project  and  even  when  prompted  by  further  government  support, 
failed  to  hold  up  its  end  of  the  partnership.  Another  respondent  felt  that  the  CRDA 
relationship,  because  it  is  not  binding,  causes  the  outside  partner  to  make  work  covered 
under  the  CRDA  the  lowest  of  its  priorities  when  they  had  additional  projects  going  on. 
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The  lessons  learned  from  the  unsuccessful  CRD  As  reinforced  the  idea  of  doing 
technology  transfer  smarter  in  the  future.  The  conventional  wisdom  is  that  as  long  as  the 
outside  partner  is  committed  to  the  technology,  the  other  benefits  will  come  to  WL 
naturally.  Developing  CRD  As  only  for  the  sake  of  increasing  the  number  of  CRD  As 
results  in  no  real  benefits  to  the  government  and  often  will  lead  to  worker  frustrations  and 
a  waste  of  time  and  resources. 

The  most  common  themes  from  the  lessons  learned  in  the  successful  CRD  As  are 
listed  below. 

a.  Outside  partner  and  WL  established  a  strong  working  relationship  with  one 
another;  CRDA  was  a  cooperative  effort  between  the  two  parties; 

b.  Outside  partner  had  a  genuine  interest  in  the  technology  that  went  beyond  a 
profit  motive; 

c.  Outside  partner  foimd  out  about  the  technology  through  informal  methods 
such  as  word  of  mouth  or  personal  contacts; 

d.  Outside  partner  usually  approached  WL  or  expressed  interest  about  a 
technology-market  pull  was  more  common  to  successful  CRD  As; 

e.  Outside  partner  had  an  in-depth  knowledge  of  the  technology  covered  imder 
the  CRDA. 

Most  of  the  respondents  pointed  out  that  the  success  of  a  given  CRDA  is  usually 
depends  upon  the  efforts  of  the  outside  partner.  In  situations  where  the  outside  partners 
are  motivated  and  interested  in  the  technology,  the  other  benefits  in  the  form  of  revenues 
will  come  to  the  government  naturally.  Two  of  the  respondents  had  worked  on  both 
successful  and  unsuccessful  CRD  As.  They  were  able  to  offer  opinions  on  why  the 
CRD  As  succeeded  or  failed.  They  pointed  out  that  they  CRD  As  that  failed  seemed 
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doomed  from  the  start  because  the  outside  partners  did  not  seem  willing  to  put  forth  the 
effort  required  under  the  terms  of  the  CRD  A.  The  respondents  felt  that  valuable  time  and 
resources  were  used  in  developing  the  CRD  As  and  that  WL  failed  to  get  an  adequate 
return  on  their  investment. 

Summary 

During  the  interview  and  data  collection  phase  of  this  study,  one  issue  that  was 
consistently  raised  by  respondents  was  that  WL  is  now  beginning  to  effectively  focus  its 
technology  transfer  efforts.  Most  respondents  felt  that  the  early  emphasis  was  on 
developing  CRD  As  for  no  real  reason  other  than  to  bolster  the  number  of  CRD  As  within 
WL.  Respondents  now  feel  that  recent  CRDAs  are  being  developed  to  create  a  mutually 
beneficial  partnership  between  WL  and  the  outside  partner.  The  prevailing  wisdom 
within  WL  is  that  it  is  doing  technology  transfer  less  frequently,  but  much  more 
successfully  than  before.  Considering  the  high  percentage  of  benefit-producing  CRDAs, 
the  results  of  this  study  seem  to  validate  this  belief  If  WL  and  other  federal  laboratories 
are  effectively  transferring  technology  to  the  private  sector,  then  effective  measures  of 
success  are  needed.  Research  conducted  by  West  (1995)  and  others  has  indicated  that  the 
quantifiable  benefits  received  by  government  laboratories  is  an  appropriate  measure  of 
technology  transfer  success. 

The  most  important  result  from  this  exploratory  study  was  that  WL  is  in  fact 
experiencing  benefits  from  their  technology  transfer  efforts.  In  a  majority  of  the  CRDAs 
researched  in  this  study,  the  benefits  received  through  technology  transfer  efforts  could  be 
quantified  by  the  ROI  methodology  described  in  Chapter  Three. 
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The  benefits  received  through  WL  CRD  As  could  be  categorized  in  to  one  of  four 
major  categories.  More  than  half  of  the  benefits  resulting  from  WL  CRDAs  were 
categorized  as  other  nonquantifiable  benefits.  However,  it  is  important  to  note  that 
respondents  felt  that  these  benefits,  although  not  quantifiable,  were  important  to  the  WL 
mission  and  represented  real  benefit  to  the  Air  Force.  A  closer  look  at  the  benefits 
received  by  WL  revealed  that  the  most  common  specific  benefit  to  the  government  came 
in  the  form  of  resource  savings  to  WL.  Therefore,  costs  were  avoided  as  a  result  of 
technology  transfer  efforts  and  these  savings  could  be  quantified  and  used  in  the  ROI 
methodology  described  in  Chapter  Three. 

The  ROI  methodology  described  in  Chapter  Three  produced  a  wide  range  of 
estimated  benefits  to  WL.  The  estimated  mean  net  present  value  of  the  quantifiable 
benefits  received  in  the  9  revenue  producing  CRDAs  ranged  from  a  low  of  $  2,000  to  a 
high  of  $  834,574.  The  CRDAs  containing  royalty  revenue  streams  provided  the  biggest 
numbers  in  terms  of  estimated  net  present  value.  It  must  be  remembered  that  there  is  a 
great  deal  of  imcertainty  included  in  these  estimates  due  to  the  speculative  nature  of  the 
input  parameters.  However,  it  is  important  to  note  that  all  of  the  CRDAs  containing 
speculative  revenues  also  provided  more  well  defined  revenues  to  WL  in  the  form  of 
resource  savings. 

The  lessons  learned  section  demonstrated  that  the  most  important  aspect  of  a 
successful  CRDA  was  the  effort  put  forth  by  the  outside  partner  to  further  enhance  the 
technology.  In  situations  where  WL  and  the  outside  partner  had  a  cooperative 
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relationship  with  one  another  and  were  willing  to  work  towards  the  advancement  of  a 
given  technology,  the  government  was  assured  some  form  of  benefits. 

Chapter  Five  utilizes  the  results  of  this  data  and  additional  evidence  gathered 
during  the  study’s  interview  process  to  develop  some  general  recommendations  to 
improve  the  Air  Force’s  efforts  in  measuring  the  benefits  received  from  technology 
transfer  efforts.  In  addition,  it  presents  several  areas  that  surfaced  during  this  study  that 
require  further  research. 
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V.  Conclusion 


Introduction 


This  chapter  offers  the  conclusions  and  recommendations  resulting  jfrom  the 
exploratory  study  conducted  on  the  benefits  received  by  the  Air  Force  through  its 
technology  transfer  efforts.  The  conclusions  will  offer  answers  to  the  research  objectives 
outlined  in  Chapter  One  and  discussed  throughout  this  research.  The  conclusions  will 
specifically  address  the  benefits  identified  by  CRDA  POCs  during  the  probability 
encoding  interviews,  the  quantification  of  those  benefits,  the  feasibility  and  usefulness  of 
methodology  used  to  quantify  those  benefits,  and  the  lessons  learned  about  technology 
transfer  efforts  at  WL.  In  addition,  this  chapter  will  document  any  future  research  topics 
stemming  from  this  exploratory  study  on  the  benefits  received  by  the  Air  Force  as  a  result 
of  technology  transfer  activities. 

Conclusions 

Analysis  of  the  probability  encoding  data  and  the  qualitative  evidence  obtained 
through  personal  interviews  with  CRDA  POCs  reveals  that,  in  most  cases,  WL  receives 
benefits  from  its  technology  transfer  activities.  Several  conclusions  can  be  drawn  from 
the  data  collected  during  this  study. 

1 .  These  benefits  are  often  quantifiable  and  in  some  cases  represent  a 
substantial  return  on  investment  from  CRD  As  developed  at  WL. 

2.  The  benefits  obtained  by  the  Air  Force  through  its  technology  transfer 
activities  are  identifiable  and  are  consistent  with  the  benefits  expected  by 
AFMC/TTO.  However,  there  is  no  useful  method  currently  in  place  to 
evaluate  the  importance  of  the  nonquantifiable  benefits. 

3.  A  net  present  value  calculation  incorporating  the  imcertainty  of  the  future 
revenue  streams  generated  CRD  As  is  a  usable  methodology  for  the  purposes 
of  calculating  the  ROI  of  technology  transfer  activities. 
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4.  A  considerable  amount  of  variation  and  uncertainty  is  contained  in  the  ROI 
calculations  built  during  this  study.  Further  research  and  data  is  needed  in  this 
area  to  validate  this  methodology  as  a  usable  management  decision  making 
tool. 

5.  Cooperation  between  the  outside  partner  and  government  agency  is  the  key 
component  of  technology  transfer  success.  The  most  successful  technology 
transfers  occur  when  the  outside  partner  has  a  genuine  interest  in  further 
developing  a  given  technology. 

Oiiantifiahlp  Benefits  to  WL.  The  quantifiable  benefits  described  by  the  respondents 
were  numerous  and  significant  in  terms  of  revenues  to  WL.  The  most  common 
quantifiable  benefit  received  by  WL  was  in  the  form  of  resources  saved,  resulting  firom 
the  efforts  of  the  outside  partner  in  the  CRDA.  The  benefits  were  quantified  using  a  net 
present  value  calculation  that  incorporated  probability  density  functions  and  Monte  Carlo 
simulation  to  account  for  the  uncertainty  involved  with  estimating  future  revenue 
streams.  During  the  data  collection  phase  of  this  exploratory  study,  respondents  provided 
quantifiable  benefits  on  nine  of  fifteen  CRD  As.  More  importantly,  of  the  eleven  CRD  As 
that  produced  benefits,  nine  of  them  produced  quantifiable  benefits.  Of  these  nine 
revenue  producing  CRD  As,  only  one  CRDA  produced  quantifiable  benefits  that  were  less 
than  $10,000.  In  addition,  six  of  the  nine  revenue  producing  CRD  As  produced  estimated 
revenues  that  were  greater  than  $100,000.  The  data  indicates  that  a  majority  of  the 
CRDAs  developed  at  WL  are  producing  significant  revenues  for  the  Air  Force.  More 
research  is  needed  to  determine  whether  or  not  the  sample  used  in  this  study  is  an 
accurate  representation  of  the  total  population  of  technology  transfers  in  WL  or  the  Air 
Force. 
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The  data  indicates  that  revenues  totaling  more  than  one  hundred  thousand  dollars 
are  not  out  of  the  ordinary.  In  situations  where  a  CRDA  is  expected  to  result  in  a 
resource  savings  to  the  Air  Force  and  the  commerciali2ation  of  a  technology,  the 
revenues  received  can  total  several  hundred  thousands  of  dollars. 

Non-Ouantifiable  Benefits  to  WL  Resulting  from  CRDAs.  The  data  collected  during 
the  interview  process  provided  evidence  that  WL  is  experiencing  benefits  from 
technology  transfer  activities.  These  non-quantifiable  benefits  were  easily  identified  by 
CRDA  POCs  during  the  interview  process.  These  benefits  matched  up  with  the  expected 
benefits  identified  by  AFMC/TTO.  It  is  important  to  note  that  the  actual  benefits 
received  through  technology  transfer  efforts  match  up  with  the  expected  benefits.  The 
majority  of  benefits  received  by  WL  were  in  the  form  of  non-quantifiable  benefits. 

The  respondents  were  quick  to  point  out  that  these  non-quantifiable  benefits 
provided  real  value  to  the  Air  Force  in  terms  of  positive  public  relations  or  goodwill  to 
WL.  However,  problems  arise  when  attempting  to  develop  a  calculation  that  puts  these 
benefits  in  quantifiable  terms.  First,  there  is  no  generally  accepted  measure  available  to 
help  put  these  benefits  into  meaningful  quantified  terms  for  the  purposes  of  a  decision 
making  model.  There  would  also  be  a  considerable  resource  cost  in  terms  of  man  hours 
used  to  develop  a  quantified  estimate  of  a  qualitative  benefit’s  value  to  WL.  Finally,  any 
quantitative  estimate  developed  from  a  purely  qualitative  benefit  would  be  highly 
subjective  and  unlikely  to  be  free  of  bias  on  the  part  of  the  respondent  and  or  estimator. 

The  most  logical  approach  would  be  to  identify  the  potential  non-quantifiable 
benefits  expected  from  a  given  CRDA.  Once  these  benefits  are  identified,  they  can  be 
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evaluated  against  WL  technology  transfer  objectives.  The  decision  to  pursue  a  given 
technology  transfer  based  solely  on  its  potential  for  providing  intangible,  non-quantifiable 
benefits  would  be  difficult  to  justify.  However,  it  is  clear  that  intangible  benefits  provide 
real  value  to  WL  and  they  should  be  evaluated  any  time  technology  transfer  activities  are 
being  considered.  In  situations  where  technology  transfer  activities  are  competing  for 
resources  and  they  are  expected  to  provide  similar  returns  in  the  form  of  quantifiable 
benefits,  then  intangible  benefits  should  be  considered  as  a  decision-making  criteria. 

Unfortunately,  past  research  by  West  and  others  indicates  that  WL  has  no  clear 
cut  objectives  for  its  technology  transfer  efforts.  Therefore,  there  is  no  current 
benchmark  or  formal  objectives  available  that  can  be  used  to  evaluate  the  impact  of  the 
qualitative  non-quantifiable  benefits  received  by  WL.  Once  these  objectives  are 
identified,  formulated,  and  advertised  to  WL  personnel,  a  useful  set  of  criteria  can  be 
established  for  the  purposes  of  evaluating  the  nonquantifiable  benefits  received  by  WL 
through  technology  transfer  activities. 

Net  Present  Value  Methodology.  Estimates  of  the  revenues  received  by  WL  from 

technology  transfer  activities  were  calculated  using  a  net  present  value  methodology. 
Uncertainty  was  incorporated  into  the  estimates  through  the  use  of  subjective  PDFs  and 
Monte  Carlo  Simulation.  The  result  was  a  ROI  calculation  that  accounted  for  the  time 
value  of  money  and  provided  a  cumulative  density  function  to  express  the  probabilities  of 
potential  revenues  to  WL. 

The  methodology  was  easily  understood  by  all  of  the  respondents  and  the  use  of 
probability  encoding  techniques  enabled  this  researcher  to  obtain  estimated  values  for  tiie 
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input  parameters  (revenue  streams).  During  the  interview  process,  respondents  were 
helpful  in  determining  whether  or  not  it  was  reasonable  to  quantify  specific  benefits.  In 
many  cases,  benefits  were  identified,  but  not  quantified,  because  respondents  felt  that  it 
was  unreasonable  to  put  them  into  quantifiable  terms.  Steps  were  taken  to  prevent  overly 
optimistic  revenue  estimates  and  minimize  the  impact  of  bias  on  the  estimates.  However, 
it  is  this  researcher’s  conclusion  that  there  is  still  some  bias  present  in  the  ROl 
calculations.  Even  for  an  experienced  analyst,  the  probability  encoding  technique  is  a 
difficult  task.  Gathering  the  data  for  this  study  was  a  rather  complex  process  that 
required  one  on  one  interviews  with  knowledgeable  CRDA  focal  points.  The  time 
consuming  nature  of  the  interview  process,  combined  with  the  complexity  and 
subjectivity  involved  with  probability  encoding  interview  techniques,  would  make  it 
difficult  to  efficiently  implement  this  methodology  on  a  large  scale. 

The  ROl  methodology  used  in  this  study  provided  estimates  of  revenues 
generated  by  technology  transfer  activities.  In  order  to  assess  the  reasonableness  of  the 
estimates,  data  will  need  to  be  collected  on  the  actual  revenues  received  from  these 
CRD  As.  There  is  no  current  formal  process  in  place  to  collect  data  on  revenues 
generated  by  CRDA  activities.  A  more  effective  data  collection  mechanism  must  be 
implemented  at  WL  so  that  revenues  received  can  be  traced  to  the  CRDA  that  generated 
the  revenue.  Once  an  effective  data  collection  process  is  implemented  at  WL,  the  ROl 
methodology  used  in  this  study  can  be  assessed  to  evaluate  its  usefulness  and  robustness 
as  a  predictor  of  future  revenues  from  technology  transfer  efforts. 
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Variation  and  Uncertainty  in  the  ROI  Calculation.  Many  of  the  estimated  revenues 
obtained  in  this  study  were  significant,  but  high  degrees  of  uncertainty  and  variation  were 
present  in  the  estimated  values.  The  high  degrees  of  variation  in  the  estimates  make  them 
somewhat  suspect.  The  speculative  nature  of  the  parameter  inputs  used  to  estimate  future 
royalty  revenue  streams  should  lead  decision  makers  to  proceed  with  caution.  The 
revenues  received  through  resource  savings  and  work  savings  are  more  well-defined  and 
represent  a  more  immediate  and  tangible  result  to  WL  from  its  technology  transfer 
activities.  These  revenues  should  probably  be  given  more  emphasis  in  the  future  than 
what  they  were  given  in  this  exploratory  study.  This  study  showed  that  resource  savings 
are  the  most  common  benefit  received  by  WL  firom  technology  transfer  activities. 

It  is  important  to  note  that  the  degree  of  variation  in  the  ROI  calculations 
coincided  with  the  number  of  well-defined  or  speculative  inputs  contained  in  the 
estimate.  The  ROI  calculations  that  contained  only  well-defined  parameter  inputs 
provided  estimates  where  the  CDF  of  the  total  outcomes  was  expressed  as  a  tighter 
confidence  interval.  This  result  is  consistent  with  this  researcher’s  expectations. 
Therefore,  it  is  assumed  that  the  degree  of  variation  contained  in  the  ROI  estimates  of  this 
study  is  reasonable  and  is  logically  tied  to  the  amount  of  uncertainty  expressed  by  the 
respondents  about  the  parameter  inputs.  On  the  surface,  this  methodology  appears  to  be 
an  appropriate  tool  that  captures  and  accounts  for  the  xmcertainty  contained  in  an 
estimate. 

Lessons  Learned.  The  lessons  learned  from  this  exploratory  study  showed  that 
cooperation  between  the  outside  partner  and  the  government  agency  was  the  most  likely 
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indicator  of  technology  transfer  success  or  failure.  The  most  common  trait  associated 
with  successful  technology  transfer  was  the  willingness  of  the  outside  partner  to  further 
develop  and  enhance  the  technology  covered  under  a  given  CRDA.  Respondents  felt  that 
the  unsuccessful  CRD  As  seemed  doomed  form  the  start.  They  also  pointed  out  that 
doing  fewer  technology  transfers  with  outside  partners  that  are  committed  to  further 
development  of  a  given  technology  is  probably  the  best  possible  way  to  maximize  the 
government’s  return  on  its  technology  transfer  efforts.  There  is  no  formal  screening 
process  in  place  to  help  personnel  at  WL  screen  prospective  outside  partners  based  on  the 
amoimt  of  effort  or  resources  they  are  willing  to  put  towards  the  enhancement  of  the 
technology  transferred  to  them. 

Managerial  Implications 

The  most  common  quantifiable  benefit  received  at  WL  from  its  technology 
transfer  efforts  are  work  savings  and  cost  avoidance  resulting  from  the  efforts  of  the 
outside  partner.  In  order  to  better  focus  its  technology  transfer  activities,  managers  at  WL 
could  attempt  to  identify  potential  areas  where  work  savings  could  be  achieved  with  the 
help  of  an  outside  partner.  Once  potential  work  savings  and  cost  avoidance  opportunities 
are  identified,  potential  outside  partners  could  be  screened  based  on  there  ability  to 
perform  the  identified  work  and  their  willingness  to  further  advance  the  technology 
covered  under  the  terms  of  the  CRDA.  Screening  of  outside  partners  could  potentially  go 
a  long  way  in  better  focusing  WL  technology  transfer  activities.  Identification  of 
potential  work  savings  and  screening  of  potential  outside  partners  could  help  WL 
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improve  its  technology  transfer  focus  and  increase  quantifiable  benefits  to  the 
government. 

WL  is  currently  experiencing  both  quantifiable  and  intangible  benefits  from  its 
technology  transfer  activities.  However,  it  unclear  as  to  which  type  of  benefits  are  most 
important  to  the  WL  mission.  In  order  to  assess  the  effectiveness  of  its  technology 
transfer  activities,  managers  need  to  develop  clear  technology  transfer  objectives.  The 
accumulation  of  quantifiable  benefits  through  CRD  As  is  probably  the  most  effective 
measure  of  transfer  success.  Unfortunately,  that  does  not  take  into  account  the  numerous 
and  very  real  intangible  benefits  that  are  currently  being  received.  The  decision  to 
allocate  resources  towards  technology  transfer  activities  based  solely  on  the  potential  for 
intangible  benefits  would  probably  be  difficult  to  justify.  A  system  is  needed  where 
decision  on  technology  transfer  activities  can  be  based  on  the  expected  quantifiable 
benefits  received  from  a  given  transfer.  Once  this  system  is  developed,  the  expected 
intangible  benefits  could  be  used  as  a  decision-making  criteria  in  situations  where 
competing  transfer  activities  are  expected  to  yield  similar  quantifiable  benefits. 

Currently  a  CRDA  is  considered  successful  as  long  as  it  provides  some  type  of 
benefits  to  WL.  Unfortunately,  the  lack  of  centralized  data  collection  and  reporting 
procedures  have  caused  several  successful  CRDAs  to  go  unrecognized.  The  collection 
and  reporting  of  data  on  technology  transfer  activities  is  left  to  the  CRDA  points  of 
contact  and  data  on  the  revenues  received  from  CRDAs  is  not  maintained  in  a  centralized 
database.  Maintaining  a  database  on  the  benefits  received  from  CRDAs  at  WL  would 
give  managers  an  excellent  tool  that  could  be  used  to  quickly  identify  technology  transfer 
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successes  and  failures.  The  lack  of  a  centralized  data  collection  and  reporting  process 
makes  the  evaluation  of  technology  transfer  activities  a  difficult  and  cumbersome  task. 

Recommendations  for  Follow-on  Research 

The  data  collected  during  this  study  demonstrates  that  the  nonquantifiable  benefits 
experienced  by  WL  are  numerous  and  represent  real  value  to  the  Air  Force. 

Unfortunately,  WL  has  no  formally  established  technology  transfer  objectives.  A  clear 
set  of  objectives  would  help  decision  makers  to  evaluate  the  value  of  nonquantifiable 
benefits  resulting  firom  CRDA  activities.  Several  of  the  nonquantifiable  benefits 
documented  in  this  research  would  provide  a  solid  foundation  that  could  be  used  as  a 
starting  block  to  build  formal  objectives  for  WL  technology  transfer  efforts.  Examples  of 
significant  nonquantifiable  benefits  include  the  dissemination  of  technology,  public 
relations  and  goodwill,  enhancements  to  WL  developed  technology,  and  improvements  in 
the  morale,  expertise  and  knowledge  of  WL  personnel.  Any  expert  would  agree  that  these 
benefits,  although  difficult  to  measure,  could  be  used  as  qualitative  factors  when 
analyzing  the  return  on  technology  transfer  activities.  By  developing  formal  objectives 
for  technology  transfer  efforts,  researchers  will  be  able  to  assess  the  value  of  technology 
transfer  nonquantifiable  benefits. 

Quantifiable  benefits  were  identified  in  nine  of  the  15  CRD  As  evaluated  in  this 
study.  Additional  data  would  be  needed  to  determine  whether  this  sample  was 
representative  of  the  total  CRDA  population  within  WL.  CRD  As  were  selected  randomly 
for  this  data  set  and  the  benefits  were  analyzed  using  a  ROI  methodology.  A  general 
survey  or  questionnaire  could  be  to  disseminated  to  all  CRDA  POCs  in  order  to 
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determine  the  total  number  of  CRDAs  that  are  providing  some  benefits  to  WL.  This 
survey  would  provide  researchers  with  an  accurate  percentage  of  benefit  versus  non¬ 
benefit  producing  CRDAs.  Once  the  total  benefit  producing  CRDAs  are  identified  they 
could  be  evaluated  more  closely  to  determine  their  actual  revenues  or  benefits. 

At  this  time  there  is  still  no  formal  method  in  place  to  effectively  measure  the 
effectiveness  of  Air  Force  technology  transfer  efforts.  Evidence  suggest  that  the  success 
or  failure  of  technology  transfer  lies  with  the  efforts  of  the  outside  partner.  At  this  time, 
there  are  no  formal  screening  procedures  in  place  to  effectively  screen  prospective  CRDA 
partners.  A  formal  approach  to  screening  CRDA  partners  could  result  in  the  Air  Force 
selecting  outside  firms  that  are  genuinely  interested  and  willing  to  put  forth  the  effort 
required  to  enhance  a  given  technology. 

Future  Research 

Anecdotal  evidence  suggests  that  WL  is  transferring  technology  more  effectively 
now  than  ever  before.  As  a  result,  it  is  hypothesized  that  quantifiable  benefits  are  more 
common  in  recent  CRDAs.  An  approach  that  could  be  used  to  study  this  question  further 
would  be  to  evaluate  a  random  sample  of  CRDAs  from  1995  against  a  random  sample  of 
CRDAs  from  1993  to  determine  if  WL  is  in  fact  receiving  more  benefits  from  recent 
CRDA  activities. 

This  study  pointed  out  that  there  are  traits  common  to  revenue  producing  CRDAs. 
Additional  research  could  focus  specifically  on  a  larger  sample  of  revenue  producing 
CRDAs.  By  focusing  exclusively  on  revenue  producing  CRDAs,  a  set  of  well  defined 
characteristics  can  be  established.  The  ROI  can  be  calculated  for  each  individual  revenue 
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producing  CRD  A.  These  CRDA  characteristics  and  revenues  could  then  be  compiled 
into  a  data  set  and  used  to  build  a  parametric  models  to  predict  future  CRDA  revenues 
based  on  their  characteristics. 

Research  conducted  at  the  WL  technology  transfer  office  suggests  that  an 
effective  measure  of  technology  transfer  success  would  be  the  level  of  effort  put  forth  by 
the  outside  partners.  The  evidence  provided  by  respondents  in  this  study  suggest  that  the 
outside  partner  is  the  driving  force  behind  CRDA  success.  Therefore,  the  measurement 
of  outside  partner  effort  as  a  means  of  determining  technology  transfer  success  appears 
reasonable  and  logical.  Additional  research  in  this  area  could  provide  an  effective 
measure  of  transfer  success. 


73 


Appendix  A.  Interview  Form 


Cooperative  Research  and  Development  Agreement  (CRD A)  Point  of 

Contact  Interview  Form 


CRDA  Tracking  Number : 

Company  Involved  : 

Point  of  Contact/Qffice  Svm/Phone  #  :  _ _ _ _ 

Identiflcation  and  Specification  of  benefits  described  in  the  CRDA: 
Identify  All  the  expected  benefits  received  from  the  CRDA: 


Provide  a  quantified  estimate  of  each  benefit  mentioned  above  in  terms  of  dollars 
saved/received,  man  hr.  saved,  or  time  saved. 


Identification  and  Specification  of  emergent  benefits  not  described  in  the  CRDA: 

Identify  all  of  the  benefits  received  through  CRDA  relationship  that  were  not  originally 
spelled  out  in  the  agreement: 


Provide  a  quantified  estimate  (if  possible)  for  each  benefit  mentioned  above  in  terms  of 
dollars  saved/received,  man  hr.  saved,  or  time  saved. 


Identify  any  nonquantifiable  benefits  received  by  the  Air  Force  as  a  result  of  the 
CRDA: 


Identify  any  lessons  learned  about  the  CRDA  process  during  the  development  of  this 
CRDA 
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Appendix  B.  CRDA  Benefits  Data 


CRDA  Tracking  Number  :  95-263-WL-Ol 
Company  Involved  :  Butler  County  Engineers 
Point  of  Contact/Office  Svm/Phone  #  : 

Purpose:  Provide  cost  effective  rehabilitation  scheme  that  uses  advanced  composite 
materials  to  reinforce  ailing  infrastructure. 

Identify  All  the  expected  benefits  received  from  the  CRDA: 

1 .  Wide  spread  use  of  composite  materials  will  result  in  cost  savings  to  the  government- 
high  production  rate  of  composite  materials  would  spread  out  overhead  costs  and  reduce 
total  cost  to  customers. 

—  These  benefits  not  yet  realized— 10  -  20  year  time  frame  before  nationwide 
implementation  of  this  technology  would  take  place— additional  interest  has  already  been 
generated  in  the  state  of  Ohio. 

2.  Royalties  would  be  paid  to  the  government  (WL)  as  a  result  of  this  technology  being 
implemented— patent  has  been  written  for  the  process  developed  under  this  CRDA 

3.  Improvements  in  national  infrastructure  and  better  use  of  taxpayer  dollars 

—  Very  real  benefit  that  will  be  realized  in  the  future  as  advanced  composite 
materials  are  used  to  repair  our  nation’s  infrastructure.  The  use  of  more  durable  and  cost 
effective  advanced  composite  materials  will  provide  cost  savings  to  the  government  as 
roadways  and  bridges  are  rebuilt  and  reinforced.  Annual  future  costs  savings  to  our 
nation  could  be  in  the  billions. 

Provide  a  quantified  estimate  of  each  benefit  mentioned  above  in  terms  of  dollars 
saved/received,  man  hr.  saved,  or  time  saved. 

1.  Patent  currently  being  developed  for  the  vacuum  bag  processing  to  externally 
reinforced  structural  members  with  advanced  composites. 

-  The  potential  cash  reimbursements  to  WL  have  been  estimated  in  the  millions; 
however,  these  revenues  are  not  expected  to  be  realized  for  another  10  to  20  years. 

Identify  all  of  the  benefits  received  through  CRDA  relationship  that  were  not 
originally  spelled  out  in  the  agreement: 


1.  All  composite  vehicle  bridge  built  by  Lockheed  may  be  used  at  a  Butler  Coxmty  site 
this  summer— providing  a  free  test  site  to  a  government  contractor  to  prove  and  validate 
this  technology. 
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2.  Ohio  Department  of  transportation  CRD  A  being  written  and  developed. 

3.  Accelerating  the  progress  of  this  technology  through  WL  participation  with  other  govt, 
agencies— WL  used  as  a  neutral  objective  third  party;  knowledgeable  scientists  and 
researchers  rather  than  private  vendors  are  advocating  the  use  of  this  technology,  helping 
it  achieve  widespread  acceptance  as  a  viable  solution  for  improving  our  nation’s 
infrastructure. 

Provide  a  quantified  estimate  (if  possible)  for  each  benefit  mentioned  above  in  terms  of 
dollars  saved/received,  man  hr.  saved,  or  time  saved. 


Identify  any  nonquantifiable  benefits  received  by  the  Air  Force  as  a  result  of  the 
CRDA: 

1 .  Good  will  and  publicity  to  WL  which  resulted  in  credibility  being  added  to  the 
technology  of  composites. 

2.  Press  coverage:  Articles  in  magazines  and  journals,  describing  how  the  technology 
was  passed  on  to  the  private  sector  (ex.  Civil  Engineering  article  March  1996) 

3.  Tours  of  high  level  personnel:  Senators,  Congressman  and  high  ranking  military 
leaders  (ex  Senator  John  Gleim) 

Identify  any  lessons  learned  about  the  CRDA  process  during  the  development  of  this 
CRDA 

1 .  Both  partners  were  eager  and  motivated  to  do  the  CRDA— the  were  willing  to  put  forth 
the  effort  to  make  the  agreement  work  out. 

2.  Process  started  with  general  discussions  and  a  small  scale  proof  of  concept  work 
before  the  CRDA  was  even  written  and  signed. 
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CRDA  Tracking  Number  :  94-173-'WL-01 
Company  Involved  :  Firefox  Industries 


Point  of  Contact/Office  Sym/Phone  #  : 

Purpose:  To  develop  an  understanding  of  fire  suppression  systems  extinguishing  fire  on 
high  speed  vehicles. 

Identify  All  the  expected  benefits  received  from  the  CRDA: 

1.  Royalty  free  use  for  govt,  purposes  in  the  imderstanding  gained  from  the  tests 

2.  Compensation  for  use  of  wind  tunnel  for  testing  and  technical  support 

-  Data  on  (Halon  vs.  water  and  oils)  generated  as  a  result  of  the  test-data  informative 
but  not  necessarily  useful  in  an  operational  military  setting 

Provide  a  quantified  estimate  of  each  benefit  mentioned  above  in  terms  of  dollars 
saved/received,  man  hr.  saved,  or  time  saved. 

2.  $45,000  paid  to  WL  for  use  of  test  facilities  and  manpower 

Identify  all  of  the  benefits  received  through  CRDA  relationship  that  were  not 
originally  spelled  out  in  the  agreement: 

1.  Partnership  for  a  new  generation  of  vehicles,  collaborative  effort  between  govt,  and 
automotive  industry,  was  aided  by  this  CRDA  in  that  it  demonstrated  a  willingness  by 
WL  to  work  with  outside  private  partners. 

Identify  any  nonquantifiable  benefits  received  by  the  Air  Force  as  a  result  of  the 
CRDA: 

-  Free  publicity  resulting  from  the  success  of  this  CRDA 

-  De-my  stifled  the  CRDA  process-the  Firefox  CRDA  demonstrated  that  CRD  As  are  an 
effective  and  viable  means  of  collaborating  with  outside  partners.  As  a  result  of  the 
Firefox  success,  other  companies  are  seeking  WL  expertise  and  are  willing  to  use  CRD  As 
as  the  mechanism  to  build  new  products  and  services. 

-  WL  personnel  gained  extensive  knowledge  about  the  formulation  of  CRD  As 

Identify  any  lessons  learned  about  the  CRDA  process  during  the  development  of  this 
CRDA 
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CRDA  Tracking  Number  :  93-336-WL-Ol 


Company  Involved  ;  Cold  Jet 
Point  of  Contact/Office  Svm/Phone  #  : 

Purpose:  Create  a  capability  to  analyze  and  improve  the  design  of  nozzles  for 

aerodynamic  particle  acceleration  for  dry  ice  pellet  blasts  surface  cleaning  and 
preparation  equipment. 

Identify  All  the  expected  benefits  received  from  the  CRDA: 

No  benefits  were  realized  from  this  CRDA;  expected  benefits  were  as  follows: 

-  Looking  to  expand  the  capability  of  CFD  and  gain  additional  nozzle  design  expertise 

-  Detailed  test  data 

-  Royalty  fees 

Provide  a  quantified  estimate  of  each  benefit  mentioned  above  in  terms  of  dollars 
saved/received,  man  hr.  saved,  or  time  saved. 


Identify  all  of  the  benefits  received  through  CRDA  relationship  that  were  not 
originally  spelled  out  in  the  agreement: 


Identify  any  nonquantifiable  benefits  received  by  the  Air  Force  as  a  result  of  the 
CRDA: 

-  WL  personnel  gained  knowledge  about  the  CRDA  process  during  the  development  of 
the  Cold  Jet  CRDA 

Identify  any  lessons  learned  about  the  CRDA  process  during  the  development  of  this 
CRDA 

-  Goals  and  expectations  for  the  project  were  to  aggressive  at  the  outset 

-  Outside  company  never  provided  the  resources  to  work  on  the  project— no  dedicated 
staff  to  work  on  the  project. 

-  The  outside  partner  initiated  the  contact  but  was  not  ready  to  commit  to  the  project 

-  The  govt,  prompted  the  outside  partner  on  several  occasions  by  offering  additional 
services  without  cooperation  from  the  outside  partner 
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CRDA  Tracking  Number  :  95-222-WL-Ol 
Company  Involved  :  Systran  Corporation 
Point  of  Contact/Office  Sym/Phone  #  : 

Purpose:  Transition  WL/FPIG  software  into  a  product  for  sale  in  the  private  sector 
(Multi  Plane  Instrumentation  Software) 

Identify  All  the  expected  benefits  received  from  the  CRDA: 

No  benefits  realized  in  this  CRDA  due  to  lack  of  effort  by  outside  partner;  expected 
benefits  were  as  follows: 

1 .  Supporting  the  development  of  a  software  library  that  will  be  maintained  and 
enhanced  by  the  collaborator  for  use  in  military  and  other  govt,  operations. 

2.  Govt,  will  receive  free  software  packages  and  user’s  manuals 

3.  the  Unit  will  earn  a  royalty  on  the  licensing  of  these  improvements  and  inventions  by 
the  collaborator. 

Provide  a  quantified  estimate  of  each  benefit  mentioned  above  in  terms  of  dollars 
saved/received,  man  hr.  saved,  or  time  saved. 


Identify  all  of  the  benefits  received  through  CRDA  relationship  that  were  not 
originally  spelled  out  in  the  agreement: 


Identify  any  nonquantifiable  benefits  received  by  the  Air  Force  as  a  result  of  the 
CRDA: 


Identify  any  lessons  learned  about  the  CRDA  process  during  the  development  of  this 
CRDA 
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CRDA  Tracking  Number  :  95-170-WL-01 

Company  Involved  :  Metrolaser 
Point  of  Contact/Office  Svm/Phone  #  ; 

Purpose:  Develop  and  evaluate  OSS  sources  to  be  used  in  the  measurement  of 
aerodynamic  parameters 

Identify  All  the  expected  benefits  received  from  the  CRDA: 

Due  to  lack  of  effort  by  the  outside  partner,  the  expected  benefits  were  never  realized. 
Expected  benefits  were  as  follows: 

1 .  Govt,  would  have  access  to  new  and  beneficial  OSS  sources 

2.  Royalty  free  use  of  new  OSS  sources 

Provide  a  quantified  estimate  of  each  benefit  mentioned  above  in  terms  of  dollars 
saved/received,  man  hr.  saved,  or  time  saved. 


Identify  all  of  the  benefits  received  through  CRDA  relationship  that  were  not 
originally  spelled  out  in  the  agreement: 


Identify  any  nonquantifiable  benefits  received  by  the  Air  Force  as  a  result  of  the 
CRDA: 


Identify  any  lessons  learned  about  the  CRDA  process  during  the  development  of  this 
CRDA 

-  CRDA  not  binding  to  the  outside  partner;  therefore,  project  covered  in  the  CRDA 
became  the  lowest  priority  for  the  outside  partner 

-  Agreement  developed  from  two  earlier  SBIRs  (CRDA  seemed  logical  but  outside 
partner  had  to  many  other  projects  going  on) 

-  Outside  partner  motivated  strictly  by  profit;  they  did  not  seem  willing  to  put  forth  the 
effort  required  to  make  the  CRDA  work 

-  Backwards  from  a  traditional  CRDA  in  that  the  outside  partner  was  going  to  develop 
the  technology  and  the  govt,  was  going  to  provide  the  test  and  evaluation  on  the 
developed  technology 
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CRDA  Tracking  Numher  :  95-004-WL-01 
C.onipanv  Involved  :  Applied  sciences  Incorporated 
Point  of  Contact/Office  Svm/Pliniie  ti  ; 

Purpose:  Develop  a  better  VG  carbon  fiber  reinforced  composite  system  for  use  in  Air 
Force  and  commercial  structures  and  bearings 

Identify  All  the  expected  benefits  received  from  the  CRDA: 

1 .  Development  of  a  new  process  to  make  composites  with  VGCF  (cooperative  R&D 
effort) 

2.  Outside  partner  providing  raw  materials  in  the  form  of  VGCF-raw  materials 
inexpensive;  therefore,  no  real  cost  avoidance  to  the  govt. 

3.  Royalty  on  the  process  developed  by  WL  personnel-up  to  5  years  after  CRDA  is 
closed  4%  of  gross  sales 

Provide  a  quantified  estimate  of  each  benefit  mentioned  above  in  terms  of  dollars 
saved/received,  man  hr.  saved,  or  time  saved. 


Identify  all  of  the  benefits  received  through  CRDA  relationship  that  were  not 
originally  spelled  out  in  the  agreement: 

1 .  Additional  testing  through  points  of  contact  done  at  no  costs  to  WL-samples  sent  to 
ASI  and  tested  at  Ohio  university  (total  costs  avoided:  $2,000) 

*  Three  to  five  samples  tested  at  an  average  cost  per  test  of  $700 

2.  Transfer  of  knowledge  with  private  industries  (ex  Goodyear,  G.M.)  through 
participation  in  meetings 

3 .  Sparked  interest  in  other  DoD  contractors  (building  industrial  base  for  these 
composites) 

-  Could  potentially  decrease  the  costs  of  composites  made  from  these  fibers 

-  Composites  with  similar  properties  at  reduced  costs-composites  currently  being 
used  for  thermal  management  for  electronic  components  costs  about  $1,500  per  poimd 
while  costs  of  new  composites  are  estimated  at  $3  per  pound 

Identify  any  nonquantiflable  benefits  received  by  the  Air  Force  as  a  result  of  the 
CRDA: 

Identify  any  lessons  learned  about  the  CRDA  process  during  the  development  of  this 
CRDA 

1 .  Great  relationship  with  outside  partner— weekly  informal  communication— foreign 
sources  of  expertise  have  been  brought  in  by  outside  partner 
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2.  ASI  approached  WL  with  the  idea  of  developing  this  CRDA 

rRDA  Tracking  Number  ;  94-241-WL-02 

rnmpanv  Involved  ;  Systems  Research  Laboratory 
Point  of  Contact/OfTice  Svm/Phone  #  : 

Purpose;  Development  of  the  Automatic  Scan-Plan  Generation  System 

Identify  All  the  expected  benefits  received  from  the  CRDA: 

The  benefits  spelled  out  in  the  CRDA  were  never  realized  due  to  lack  of  effort  by  the 
outside  partner;  the  following  benefits  were  spelled  out  in  the  CRDA: 

1 .  Broad  and  rapid  dissemination  of  technology 

2.  Royalties  expected  through  he  dissemination  of  the  technology 

3.  Long  term  costs  savings  anticipated  for  future  development  of  scan  plans  for 
additional  engine  parts 

4.  development  of  an  electronic  information  link  between  WL  and  the  outside  partner 


Provide  a  quantified  estimate  of  each  benefit  mentioned  above  in  terms  of  dollars 
saved/received,  man  hr.  saved,  or  time  saved. 


Identify  all  of  the  benefits  received  through  CRDA  relationship  that  were  not 
originally  spelled  out  in  the  agreement: 


Identify  any  nonquantifiable  benefits  received  by  the  Air  Force  as  a  result  of  the 
CRDA: 

-  Learned  to  automate  the  development  of  scan  plans 

Identify  any  lessons  learned  about  the  CRDA  process  during  the  development  of  this 
CRDA 

-  The  only  real  motive  for  the  outside  partner  was  to  obtain  the  use  of  a  developed 
technology-outside  partner  put  forth  no  effort  to  license  an  already  patented  technology 

-  SRL  had  no  real  interest  in  improving  productivity  of  the  technology  itself 
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CRDA  Tracking  Number  :  94-12-WL-Ol 

Company  Involved  :  Technosoft  Incorporated 


Point  of  Contact/Office  Svm/Phone  #  : 

Purpose:  Development  of  a  memory  driven  design  system  capable  of  integrating 
product  descriptions  v^th  materials  and  processing  knowledge. 

Identify  All  the  expected  benefits  received  from  the  CRDA: 

1 .  Unusually  broad  and  rapid  dissemination  of  its  technology 

—  Effectively  licensing  some  of  the  patents  and  making  modules  developed  from  the 
technology  part  of  the  outside  partner’s  system-5%  royalty  to  WL  on  the  sale  of  each 
module 

2.  Further  enhancement  of  this  technology  through  involvement  of  industrial  design 
system  houses  and  tool  vendors-using  the  technology  in  private  industry 


Provide  a  quantified  estimate  of  each  benefit  mentioned  above  in  terms  of  dollars 
saved/received,  man  hr.  saved,  or  time  saved. 


Identify  all  of  the  benefits  received  through  CRDA  relationship  that  were  not 
originally  spelled  out  in  the  agreement: 

1 .  The  use  of  additional  technologies  by  outside  partners 


Identify  any  nonquantifiable  benefits  received  by  the  Air  Force  as  a  result  of  the 
CRDA: 

-  Public  relations:  Technosoft  has  advertised  the  fact  that  the  technology  they  are 
marketing  was  developed  at  WL 

Identify  any  lessons  learned  about  the  CRDA  process  during  the  development  of  this 
CRDA 
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CRDA  Tracking  Number :  94-132-WL-Ol 


Company  Involved  :  Technology  Assessment  and  Transfer  Incorporated 
Point  of  Contact/Oflice  Svm/Phone  #  : 

Purpose:  The  identification  of  dual  use  partners  for  the  commercialization  and  or 
enhancement  of  molecular  beam  epitaxy  (MBE)  thin  film  disposition  technology. 

Identify  All  the  expected  benefits  received  from  the  CRDA: 

1 .  Broad  and  rapid  dissemination  of  molecular  beam  epitaxy  (MBE)  technology— outside 
partner  performing  the  marketing  function  to  get  the  technology  to  private  industries 

*  One  man  year  saved  marketing  the  technology  to  industrial  users  (60  k ) 

2.  Identifying  dual  use  partners  for  the  MBE  technology  and  presenting  these  sites  to  WL 

Provide  a  quantified  estimate  of  each  benefit  mentioned  above  in  terms  of  dollars 
saved/received,  man  hr.  saved,  or  time  saved. 


CRDA#  94-132-WL-01 

Company  :  Technology  Assessment  and  Transfer  Corporation 

Expected  Benefits  1996  1997  1998  1999  2000 

Marketing  of  MBE  $  46,599  0  0 

- 1  man  year  effort 
Cost  Avoidance 

Present  Value  $  43,551 


Identify  all  of  the  benefits  received  through  CRDA  relationship  that  were  not 
originally  spelled  out  in  the  agreement: 

1 .  Assist  and  guide  WL  in  the  development  of  MBE  technology  training  manuals 

-  Authoring  the  current  manual  and  rewriting  the  manual  for  commercial  use 

-  Outside  partner  provided  additional  insight  into  the  needs  of  the  end-users 

*  one  person  used  for  this  effort 

Identify  any  nonquantifiable  benefits  received  by  the  Air  Force  as  a  result  of  the 
CRDA: 

1 .  This  CRDA  allowed  WL  to  do  technology  transfer  between  federal  agencies: 
University  of  Maryland  working  exclusively  with  the  National  Security  Agency  was  able 
to  gain  insight  into  the  use  of  MBE  technology 

Identify  any  lessons  learned  about  the  CRDA  process  during  the  development  of  this 
CRDA 

1 .  Outside  partner  found  out  about  the  technology  through  SIBR  relationship  formed 
prior  to  the  CRDA-TA&T  approached  WL  about  the  technology 
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Crystal  Ball  Report 

Simulation  started  on  7/17/96  at 
20:18:55 

Simulation  stopped  on  7/17/96  at 
20:19:05 


Forecast:  Cost  Avoidance  Present  Value 
Summary 

Display  Range  is  from  $27,500  to  $57,500  dollars 
Entire  Range  Is  from  $28,775  to  $55,627 
dollars 

After  1,000  Trials,  the  Std.  Error  of  the  Mean  is  $182 


Statistics 

V9.I.U9. 

Trial 

1000 

s 

Mea 

$43,551 

n 

Median  (approx.) 

$44,206 

Mode  (approx.) 

$42,873 

Standard 

$5,758 

Deviation 

Variance 

$33,156,607 

Skewness 

-0.31 

Kurtosis 

2.38 

Coeff.  of 

0.13 

Variability 

Range  Minimum 

$28,775 

Range  Maximum 

$55,627 

Range  Width 

$26,852 

Mean  Std.  Error 

$182.09 

Cell  B9 


Forecast:  Cost  Avoidance  Present  Value 

Frequency  Chart  1 ,000  T rials  Shown 


n 

xn 

c 

n> 

=3 
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Forecast:  Cost  Avoidance  Present  Value  (cont'd) 


Percentiles: 


Percentile 

dollars  (approx.) 

0% 

$28,775 

10% 

$35,348 

20% 

$38,144 

30% 

$40,523 

40% 

$42,537 

50% 

$44,206 

60% 

$45,666 

70% 

$47,020 

80% 

$48,713 

90% 

$51,018 

100% 

$55,627 

Assumption:  Marketng  of  MBE 

Triangular  distribution  with  parameters: 

Marketing  < 

Minimum 

Likeliest 

Maximum 


$30,000 

$50,000 

$60,000 


Selected  range  is  from  $30,000 to  $60,000 
Mean  value  in  simulation  was  $46,599 


Cell:  B6 
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CRDA  Tracking  Number  :  95-335- WL-01 
Company  Involved  :  Holman  Plating 
Point  of  Contact/Office  Sym/Phone  #  : 

Purpose:  Reduce  the  cost  and  improve  the  quality  of  Titanium  Nitrate  (TIN)  technology 
through  involvement  with  outside  partner.  The  developed  TIN  coating  will  be  available 
for  Air  Force  use 

Identify  All  the  expected  benefits  received  from  the  CRDA: 

1 .  Govt,  will  explore  the  implementation  of  in-site  plasma  control  and  electric  arc 
physical  vapor  disposition  processes  while  simultaneously  allowing  a  reduction  in 
existing  military  component  coating  costs. 

-  Some  of  the  parts  that  have  been  coated  are  already  flying  on  operational  aircraft- 
process  refinement  has  already  been  achieved 

*  Added  instrumentation  involved  in  the  refinement  process  would  have  costs  the  Air 
Force  100k  had  we  not  used  CRDA 

2.  WL  paid  a  consulting  fee  for  99  hr.  of  time  spent  by  WL  persotmel“$2 1,000  paid  for 
99  hr.  of  work 

-  work  included  set-up,  equipment  implementation,  system  integration,  sample 
generation,  XPS  analysis,  and  post  analysis 


Provide  a  quantified  estimate  of  each  benefit  mentioned  above  in  terms  of  dollars 
saved/received,  man  hr.  saved,  or  time  saved. 

CRDA#  95-335-WL-01 

Company :  Holman  Plating 

Expected  Benefits  1996  1997  1998 

Consulting  Fee  $  21 ,000 

Cost  Avoidance  $103,252 

-  Added  instrumentation 

Total  Revenues  $124,252 

Present  Value  Revenue  $116,123 

Identify  all  of  the  benefits  received  through  CRDA  relationship  that  were  not 
originally  spelled  out  in  the  agreement: 

1 .  Parts  coated  with  the  process  may  lead  to  improved  life  and  durability  of  our  aircraft. 

Identify  any  nonquantifiable  benefits  received  by  the  Air  Force  as  a  result  of  the 
CRDA: 

1 .  Good  PR  for  WL  and  the  Air  Force  as  a  whole 
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Crystal  Ball  Report 

Simulation  started  on  7/17/96  at 
19:34:47 

Simulation  stopped  on  7/17/96  at 
19:34:56 


Forecast:  Present  Value  Revenue 
Summary 

Display  Range  is  from  $102,500  to  $132,500  dollars 
Entire  Range  is  from  $104,327  to  $131,061  dollars 
After  1,000  Trials,  the  Std.  Error  of  the  Mean  is  $187 


Statistics 

Value 

Trial 

1000 

s 

Mea 

$116,123 

n 

Median  (approx.) 

$115,573 

Mode  (approx.) 

$114,352 

Standard 

$5,927 

Deviation 

Variance 

$35,127,545 

Skewness 

0.29 

Kurtosis 

2.43 

Coeff.  of 

0.05 

Variability 

Range  Minimum 

$104,327 

Range  Maximum 

$131,061 

Range  Width 

$26,734 

Mean  Std.  Error 

$187.42 
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Forecast:  Present  Value  Revenue  (cont'd) 


Percentiles: 


Percentile 

dollars  (approx.) 

0% 

$104,327 

10% 

$108,471 

20% 

$110,803 

30% 

$112,644 

40% 

$114,174 

50% 

$115,573 

60% 

$117,249 

70% 

$119,102 

80% 

$121,348 

90% 

$124,823 

1 00% 

$131,061 

Assumption:  Cost  Avoidance  due  to  added  instrumentat 


Cell: 


Triangular  distribution  with  parameters: 
Minimum  $90,000 

Likeliest  $100,000 

Maximum  $120,000 


Selected  range  is  from  $90,000 to  $120,000 
Mean  value  in  simulation  was  $103,252 


Cost  Avoidance  due  to  added  Instrumentat 


*90,000  *97,500  *105,000  *112,500  *120,000 
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CRDA  Tracking  Number  :  95-075-WL-01 

Company  Involved  :  Aviation  Environmental  Compliance  Incorporated 
Point  of  Contact/Office  Sym/Phone  #  : 

Purpose:  A  design  effort  on  the  part  of  WL  to  improve  the  existing  nozzle  configuration 
and  compressed  air  system  for  (forced  air  de-icing  system)  FADS 

Identify  All  the  expected  benefits  received  from  the  CRDA: 

1.  Expanding  its  nozzle  design  experience  into  new  applications—process  currently  being 
refined  and  validated  for  commercial  and  Air  Force  use. 

2.  Royalties  expected  from  the  licensing  of  a  new  nozzle  design  (2-5  years  in  the  future 
based  on  industry  demand) 

*  $500  royalty  fee  paid  to  WL  for  every  nozzle  sold  commercially.  Sales  estimates 
will  vary  based  on  industry  demand. 

3.  Reduced  logistics  costs  associated  with  glycol  storage,  usage,  and  disposal  for  de¬ 
icing  operations 

*  Total  costs  savings  per  plane  estimated  at  $10  per  gallon  of  glycol  not  used  (this 
$10  per  gallon  of  Glycol  cost  includes  procurement,  storage,  and  disposal  costs).  A  70% 
to  90%  reduction  in  glycol  use  is  expected  if  the  new  nozzle  design  and  process  is 
validated  for  Air  Force  use.  Currently  70  -  80  gallons  are  used  per  minute  when  de-icing 
a  plane.  Estimates  for  de-icing  a  plane  range  from  15-30  minutes.  Potential  for  costs 
savings  attributed  to  the  reduction  in  glycol  use  estimated  at  $12,000  per  plane. 

Provide  a  quantified  estimate  of  each  benefit  mentioned  above  in  terms  of  dollars 
saved/received,  man  hr.  saved,  or  time  saved. 


CRDA#  95-075-WL-01 

Company  :  Aviation  Environmental  Compliance  Inc. 

Net  Present 


Expected  Benefits 

1996 

1997 

1998 

1999 

2000 

2001  Value  at  7% 

Royalty  Revenues 

$ 

$50,419 

$60,314 

$  69,992 

$  74,867 

$  74,872  ^ 

-  Rev.  per  Nozzle 

500 

500 

500 

500 

500 

500 

-  Nozzles  sold 

0 

100.8377 

120:6285 

139:9835 

149.7348 

149.7432 

Reduced  Log.  costs 

$12,159 

$  12,163 

$  12,022 

$  12,176 

$  12,136 

$  1^021 

-  Glycol  cost  per  gal 

$  10 

$  10 

$  10 

$  10 

$  10 

$  10 

-  Gallons  per  min. 

70 

70 

70 

70 

70 

70 

-  Min  to  De-ice  plane 

21.65883 

21:61457 

21:49213 

21.63273 

21.66536 

21.50181 

-  %  of  Glycol  not  used 

80% 

80% 

80% 

80% 

80% 

80% 

PV  of  Royalty  Rev. 

$  . -  ■ 

$  44,038 

$49,234 

$53,3196 

$  53,379 

$  49:890  $  249.938 

Identify  all  of  the  benefits  received  through  CRDA  relationship  that  were  not 
originally  spelled  out  in  the  agreement: 

1 .  Increase  in  the  personal  experience  of  WL  personnel 
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2.  Potential  for  future  CRD  As  with  different  technologies  (ex  spray  wash  with  glycol 
capabilities) 

Identify  any  nonquantifiable  benefits  received  by  the  Air  Force  as  a  result  of  the 
CRDA: 

1 .  Substantial  amount  of  positive  publicity  for  WL  and  Air  Force,  resulting  from  CRDA 

2.  Established  a  strong  relationship  with  Wright  Technology  Network 

Identify  any  lessons  learned  about  the  CRDA  process  during  the  development  of  this 
CRDA 

1 .  Worked  according  to  prescribed  TT  process— collaborator  questioned  WTN  about  a 
technology  and  WTN  facilitated  transfer  between  WL  and  collaborator 

2.  Outside  partner  had  an  interest  in  the  technology  that  went  beyond  profit  motivation 

3.  This  was  a  true  partnership  in  every  sense  of  the  word.  WL  consultation  actually 
saved  AEC  on  development  costs  by  recommending  that  the  proposed  test  plan  be 
simplified  and  scaled  down  to  include  only  the  necessary  tests  to  validate  and  verify  the 
FADS  process.  AEC  saved  extensive  tests  dollars  and  effort  that  would  have  beeii  spent 
without  the  help  of  WL  personnel. 

4.  There  was  a  clearly  defined  goal  for  this  CRDA— objectives  were  clear  and  obtainable 


Crystal  Ball  Report 
Simulation  started  on  7/1 6/96  at 
17:36:30 

Simulation  stopped  on  7/16/96  at 
17:37:05 


Forecast:  Royalty  Revenues  from  Nozzle  Design 
Summary 

Display  Range  is  from  $150,000  to  $350,000  Dollars 
Entire  Range  is  from  $154,870  to  $342,666  Dollars 
After  1,000  Trials,  the  Std.  Error  of  the  Mean  is  $1,120 

Statistics  Value 

Trial  1000 

s 

Mea  $249,938 

n 
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Median  (approx.) 

$249,811 

Mode  (approx.) 

$247,829 

Standard 

$35,421 

Deviation 

Variance 

$1,254,665,367 

Skewness 

0.04 

Kurtosis 

2.74 

Coeff.  of 

0.14 

Variability 

Range  Minimum 

$154,870 

Range  Maximum 

$342,666 

Range  Widtli 

$187,796 

Mean  Std.  Error 

$1,120.12 

Forecast:  Royalty  Revenues  from  Nozzle  Design 


Cell  H14 


Frequency  Chart 


1,000  Trials  Shown 


iA 

C 

CD 

S 


Forecast:  Royalty  Revenues  from  Nozzle  Design  (cont'd) 
Percentiles: 


Jentile 

Dollars  (approx.) 

0% 

$154,870 

10% 

$204,740 

20% 

$220,129 

30% 

$231,240 

40% 

$241,256 

50% 

$249,811 

60% 

$258,534 

70% 

$266,783 

80% 

$278,972 

90% 

$298,221 

100% 

$342,666 
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Assumption:  Nozzles  sold  year  2 


Normal  distribution  with  parameters: 

Mean  100.00 

Standard  Dev.  20.00 

Selected  range  is  from  -Infinity  to  -linfinity 
Mean  value  in  simulation  was  100.84 

Correlated  with: 

Nozzles  sold  year  3  (D8) 


Nozzles  sold  year  2 


0.25 


Assumption:  Nozzles  sold  year  3 


4 


Normal  distribution  with  parameters: 

Mean  120.00 

Standard  Dev.  30.00 

Selected  range  is  from -Infinity  to  -infinity 
Mean  value  in  simulation  was  120.63 

Correlated  with: 

Nozzles  sold  year  2  {C8) 

Nozzles  sold  year  4  (E8) 


Nozzles  sold  year  3 


0.25 

0.25 


» 


Assumption:  Nozzles  sold  year  4 


Normal  distribution  v\ith  parameters: 

Mean  140.00 

Standard  Dev.  40.00 

Selected  range  is  from  -Infinityto  -(Infinity 
Mean  value  in  simulation  was  139.98 

Correlated  with: 

Nozzles  sold  year  3  (D8) 

Nozzles  sold  year  5  (F8) 

Assumption:  Nozzles  sold  year  5 


Nozzles  sold  year  4 


0.25 

0.25 


Normal  distribution  with  parameters: 

Mean  150.00 

Standard  Dev.  40.00 

Selected  range  is  from  -Infinityto  -(Infinity 
Mean  value  in  simulation  was  149.73 

Correlated  with: 

Nozzles  sold  year  4  (E8) 

Nozzles  sold  year  6  {G8) 


Nozzles  sold  year  5 


30.00  80.00  150.00  210.00  270.00 


0.25 

0.25 
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Assumption:  Nozzles  sold  year  6 

Normal  distribution  with  parameters: 

Mean  150.(X) 

Standard  Dev.  40.00 

Selected  range  is  from -Infinity  to  -rtnfinit/ 
Mean  value  in  simulation  vwas  149.74 

Correlated  with: 

Nozzles  sold  year  5  (F8) 


Nozzles  sold  year  6 


90.00  tSO.OO  ZIO.OO  270,00 


0.25 
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CRDA  Tracking  Number :  95-201-WL-01 
Company  Involved  :  Paragon  Aircraft 
Point  of  Contact/Office  Sym/Phone  #  : 


Purpose:  Analyze  and  develop  the  design  of  a  single  turbo-fan  engine  aviation  aircraft 
which  has  the  following  attributes: 

-  Inlet  over  the  fuselage,  a  high  aspect  ratio  straight  wing  and  utilizes  a  composite 
structure 

-  Performance  analysis  will  refine  the  aerodynamic  characteristics  of  a  new  air 
flow  and  will  work  to  ensure  the  structural  integrity  of  the  composite  airframe 

Identify  All  the  expected  benefits  received  from  the  CRDA: 

1 .  Flight  Dynamics  Branch  gaining  increased  experience  in  CFD  and  Flutter  analysis  and 
ground  vibration  tests 

2.  Usable  test  data  derived  from  the  CRDA-test  conducted  here  at  WPAFB 

3.  Wind  tunnel  testing  still  to  be  conducted  (future  of  the  facilities  may  be  in  jeopardy 
this  CRDA  provides  additional  justification  to  keep  this  facility  operational) 

4.  Royalty  stream  expected  to  recoup  man  hr.  expended  during  this  CRDA. 

Provide  a  quantified  estimate  of  each  benefit  mentioned  above  in  terms  of  dollars 
saved/received,  man  hr.  saved,  or  time  saved. 

CRDA#  95-201  -WL-01 

Company :  Paragon  Aircraft  in  conjunction  with  WL  Flight  Dynamics  Branch 


Net  Present 


Expected  Benefits 

1996 

1997 

1998 

1999 

2000 

2001  Value  at  7% 

Royalty  Stream 

$ 

$ 

$ 

$51,622 

$  98,514 

$  37,465 

-  Aircraft  Sold 

0 

0 

0 

2.671945 

5.099084 

1.939162 

-  Rev.  per  Aircraft 

$  19,320 

$  19,320 

$  19,320 

$  19,320 

$  19,320 

$  19,320 

Payment  for  testing 

$801,417 

^  *  Fatigue  and  Static 

test  to  be  conducted 

»  at  WL  facility 

Total  Revenue 

$ 

$801,417 

$ 

$51,622 

$  98,514 

$  37,465 

Present  Value  at  7% 

$ 

$699,988 

$ 

$  39,382 

$  70,239 

S  24.964  $  834.574 

Identify  all  of  the  benefits  received  through  CRDA  relationship  that  were  not 
originally  spelled  out  in  the  agreement: 

1 .  The  CRDA  provided  an  opportunity  to  demonstrate  and  validate  existing  capabilities 
at  WL  for  a  whole  airplane 

-  Additional  breadth  gained  by  conducting  developmental  testing  on  an  entire  aircraft 
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2.  The  CRDA  helped  to  justify  the  resources  used  to  do  testing  on  the  aircraft 

3.  Insight  gained  into  the  commercial  aircraft  industry— the  feeling  was  that  WL  had  lost 
touch  with  the  civilian  aircraft  market;  this  CRDA  helped  us  get  our  foot  in  the  door  and 
gain  additional  insight  into  the  civilian  aircraft  industry 

4.  Fatigue  and  static  test  to  be  conducted  at  WL  structures  test  facility—Facility 
contractor  will  receive  payment  for  testing  conducted  at  WL  as  well  as  additional  work 
that  will  help  to  smooth  out  the  testing  schedule.  Additional  knowledge  will  be  gained  in 
the  area  of  flutter  analysis 

*  Paragon  will  pay  the  total  costs  of  static  testing  to  the  contractor  that  runs  the 
structure  test  facility. 


Identify  any  nonquantifiable  benefits  received  by  the  Air  Force  as  a  result  of  the 
CRDA: 

-  Public  relations  benefit— goodwill  towards  the  military  and  Air  Force  in  general 

-  Provides  insight  into  the  FAA  certification  process— this  will  provide  WL  the 
opportunity  to  see  another  approach  to  solving  problems 

-  Increased  capability  at  lower  costs 

Identify  any  lessons  learned  about  the  CRDA  process  during  the  development  of  this 
CRDA 

-  Mutually  beneficial  agreement  where  both  sides  pursued  the  agreement  and  are  actually 
working  to  achieve  a  desired  outcome. 


Crystal  Ball  Report 

Simulation  started  on  7/16/96 
at  15:32:38 

Simulation  stopped  on  7/16/96 
at  15:33:02 


Forecast:  Present  value  of  Paragon  CRDA 

Summary 

Display  Range  is  from  $550,000  to  $1,100,000  Dollars 
Entire  Range  is  from  $523,003  to  $1,076,825  Dollars 
After  1,000  Trials,  the  Std.  Error  of  the  Mean  is  $3,237 

Statistics  Value 


96 


Trial 

o 

1000 

o 

Mea 

n 

Median  (approx.) 

$834,574 

$843,481 

Mode  (approx.) 

$935,600 

Standard 

$102,354 

Deviation 

Variance 

$10,476,257,251 

Skewness 

-0.35 

Kurtosis 

2.60 

Coeff.  of 

0.12 

Variability 

Range  Minimum 

$523,003 

Range  Maximum 

$1,076,825 

Range  Width 

$553,822 

Mean  Std.  Error 

$3,236.70 

Cell  H14 


Forecast:  Present  value  of  Paragon  CRDA 
Frequency  Chart 


996  Trials  Shown 


c 

fS 

5 


Forecast:  Present  value  of  Paragon  CRDA  (cont'd) 


Percentiles: 


Percentile 

Dollars  (approx. 1 

0% 

$523,003 

10% 

$690,380 

20% 

$740,839 

30% 

$786,464 

40% 

$814,682 

50% 

$843,481 

60% 

$873,552 

70% 

$896,525 

80% 

$928,348 

90% 

$958,340 

100% 

$1,076,825 
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Assumption:  Aircraft  sold  year  4 


Triangular  distribution  with  parameters: 


Minimum 

1.00 

Likeliest 

3.00 

Maximum 

4.00 

Selected  range  is  from  1  .OOto  4.00 
Mean  value  in  simulation  was  2.67 


Aircraft  sold  year  4 


t.OO  1.75  2.50  3.25  4.00 


Correlated  with: 

Aircraft  sold  year  5  (F7)  0.25 

Aircraft  sold  in  year  6  (G7)  0.15 


f 


Assumption:  Aircraft  sold  year  5 


Normal  distribution  v\^th  parameters: 

Mean  5.00 

Standard  Dev.  2.00 

Selected  range  isfrom-Infinit/to  -tinfinity 
Mean  value  in  simulation  was  5.10 


Aircraft  sold  year  5 


Correlated  with: 

Aircraftsold  year4  (E7)  0.25 

Araaftsold  in  year  6  (GT)  0.25 


Assumption:  Aircraftsold  in  year  6 


Normal  distribution  vwth  parameters: 

Mean  2.00 

Standard  Dev.  1.00 

Selected  range  is  from  -Infinity  to  -rtnfinity 
Mean  value  in  simulation  was  1.94 


Correlated  with: 

Aircraftsold  year  4  (E7)  0.15 

Aircraftsold  year  5  (FT)  0.25 
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Assumption:  payment  for  static  and  fatigue  teste 


Triangular  dislribulion  with  parameters: 
Minimum  $500,000 

Likeliest  $900,000 

Maximum  $1,000,000 

Selected  range  is  from  $500,000  to  $1,000,000 
Mean  value  in  simulation  was  $801,417 


payment  for  static  and  fatigue  tests 
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CRDA  Tracking  Number  ;  94-083-WL-01 

Company  Involved  :  I.  M.  Systems  Group 
Point  of  Contact/Office  Svm/Phone  #  : 

Purpose:  Provide  commercialization  of  the  Air  Force  Acquisition  Model  technology 
software  and  effective  continued  development  of  the  AFAM 

Identify  All  the  expected  benefits  received  from  the  CRDA: 

1 .  Enhancements  to  the  AFAM  at  no  expense  to  the  govt. 

*  One  man  year  of  work  saved  under  the  terms  of  the  CRDA—estimated  at  2000 
hr.  per  year  at  a  cost  between  $47  -  $55  per  hr. 

2.  20  %  of  all  revenues  received  in  the  sale  of  AFAM  by  I.  M.  systems  (for  the  lifetime 
of  the  product) 

*  20%  of  each  unit  sold  at  a  costs  of  $80  per  unit  estimates  10,000  imit  demand  in 
other  govt,  agencies  and  an  additional  1000  units  to  contractors 

3.  Provided  access  to  the  test  bed  and  the  books  and  manuals  to  go  along  with  the  test 
bed-test  bed  developed  by  I.M.  systems  at  an  off  base  location. 

*  Manuals  estimated  between  $200  to  $400  per  year. 

Provide  a  quantified  estimate  of  each  benefit  mentioned  above  in  terms  of  dollars 
saved/received,  man  hr.  saved,  or  time  saved. 


CRDA# 

94 -083 -AS- 

01 

Company : 

1.  M.  Systems 

Net  Present 

Expected  Benefits 

Year  1  Year  2 

Year  3 

Year  4  Year  5  Value  at  7% 

One  man-yr.  work  saved 

$  102,637 

-  hr.  per  year 

$  2,001 

-  cost  per  hr. 

51.3039212 

Test  Bed  and  Manuals 

$  309 

$ 

311 

$ 

310 

$ 

309 

311 

CompuServe  Account 

$  201 

$ 

199 

$ 

200 

$ 

200 

$ 

200 

AFAM  Revenue  20%  of  GS 

$ 

15,870 

$ 

39,939 

$ 

55,808 

$ 

40,077 

-  price  per  unit 

$ 

80.00 

$ 

80.00 

$ 

80.00 

$ 

80.00 

-  estimated  units  sold 

991.868391 

2496.19093 

3481 

8,02065  : 

250^ 

CO; 
OO  ! 

O 

OO 

Total  Revenue 

$  103,148 

$ 

16,380 

$ 

40,448 

$ 

56,318 

$ 

40,588 

Present  Value  at  7% 

$  96,400 

$ 

14,307 

$ 

33,018 

$ 

42,965 

$ 

28,939  $  215.628 

Identify  all  of  the  benefits  received  through  CRDA  relationship  that  were 

not 

originally  spelled  out  in  the  agreement: 

1 .  CompuServe  account  provided  at  no  costs  to  govt. 
*  several  hundred  dollars  per  year 
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2.  CRDA  resulted  in  a  strong  relationship  between  the  govt,  and  LM.  systems 

-  Because  of  the  govt,  confidence  in  the  work  performed  by  l.M.  Systems  we  were 
able  to  offer  another  contract  to  small  business  set  aside  companies  and  avoid  the  larger 
contractors  that  would  have  most  likely  charged  the  govt,  a  higher  price  for  the  work  to 
be  performed  under  the  contract. 

*  diminished  risks  associated  with  offering  a  small  business  set  aside  contract. 

Identify  any  nonquantifiable  benefits  received  by  the  Air  Force  as  a  result  of  the 
CRDA: 

*  1 .  Improved  working  relationship  between  l.M.  Systems  and  govt,  because  l.M.  Systems 
now  had  a  stake  in  the  success  of  an  Air  Force  developed  product. 

# 

2.  The  fact  that  the  unit  has  a  CRDA  in  place  was  an  additional  benefit  because  it  proved 
that  CRD  As  are  an  effective  tool  that  can  used  by  the  Air  Force  to  transfer  technology. 

Identify  any  lessons  learned  about  the  CRDA  process  during  the  development  of  this 
CRDA 

1 .  CRD  As  should  only  go  to  companies  that  have  a  definite  interest  in  the  technology 
and  are  willing  to  put  forth  the  effort  required  to  make  the  agreement  work. 

2.  Good  working  relationship  between  the  unit  and  collaborator  made  this  CRDA  a 
cooperative  effort  where  both  parties  were  working  towards  a  common  goal. 

3.  The  outside  partner  had  an  in-depth  working  knowledge  of  the  AFAM;  they  were  able 
to  present  the  information  in  a  user  friendly  environment  through  a  personal  computer. 

4.  Realizing  that  a  CRDA  is  an  additional  marketing  tool  that  can  be  used  to  get  our 
products  to  private  industry  and  create  benefits  for  the  govt,  will  help  motivate  future 
CRDAs. 


Crystal  Ball  Report 

Simulation  started  on  7/16/96  at 
10:24:46 

Simulation  stopped  on  7/16/96  at 
10:25:33 


Forecast:  Net  Present  Value  of  AFAM  CRDA 
Summary 

Display  Range  is  from  $170,000  to  $260,000  Dollars 
Entire  Range  is  from  $171,641  to  $261,574  Dollars 
After  1,000  Trials,  the  Std.  Error  of  the  Mean  is  $458 

Statistics  Value 
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Trial 

c 

1000 

o 

Mea 

n 

$215,630 

1 1 

Median  (approx.) 

$215,948 

Mode  (approx.) 

$214,360 

Standard 

$14,492 

Deviation 

Variance 

$210,005,822 

Skewness 

0.01 

Kurtosis 

2.80 

Coeff.  of 

0.07 

Variability 

Range  Minimum 

$171,641 

Range  Maximum 

$261,574 

Range  Width 

$89,933 

Mean  Std.  Error 

$458.26 

Forecast:  Net  Present  Value  of  AFAM  CRD  A  (cont'd) 


Percentiles: 


Percentile 

Dollars  (aporox.) 

0% 

$171,641 

10% 

$196,523 

20% 

$202,989 

30% 

$207,989 

40% 

$212,068 

50% 

$215,948 

60% 

$219,119 

70% 

$223,383 

80% 

$227,645 

90% 

$234,295 

100% 

$261,574 
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Assumption:  -  hr.  per  year 


Normal  distribution  with  parameters: 

Mean  2,000.00 

Standard  Dev.  50.00 

Seiected  range  is  from  -Infinity  to  -rtnfinily 
Mean  value  in  simulation  was  2,000.57 


•  hr.  per  year 


(  Assumption:  -  cost  per  hr. 


T riangular  distribution  with  parameters: 
Minimum  47.00 

Likeliest  52.00 

Maximum  55.00 

Selected  range  is  from  47.00 to  55.00 
Mean  value  in  simulation  vwas  51.30 


-  coat  per  hr. 


Assumption:  T est  Bed  and  Manuals 


Triangular  distribution  with  parameters: 
Minimum  200.00 

Likeliest  330.00 

Maximum  400.00 


Selected  range  is  from  200.00to  400.00 
Mean  value  in  simulation  was  309.26 


Test  Bed  and  Manuals 


200.00  2S0.00  300.00  3SO.OO  400.00 


Assumption:  C9 


Triangular  distribution  with  parameters: 
Minimum  200.00 

Likeliest  330.00 

Maximum  400.00 

Selected  range  is  from  200.00to  400.00 
Mean  value  in  simulation  was  311.16 


C9 


200.00  250.00  300.00  350.00  400.00 
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Assumption:  D9 


Triangular  distribution  with  parametErs: 
Minimum  2(30.00 

Likeliest  330.00 

Maximum  400.00 


Selected  range  is  from  2(30.00 to  400.00 
Mean  value  in  simulation  was  309.64 
Assumption:  E9 


09 


200.00  250.00  300.00  3S0.00  400.00 


Triangular  disfibution  with  parameters: 
Minimum  200.0 

Likeliest  330.0 

Ma)amum  400.0 


Selected  range  is  from  200.00to  400.00 
Mean  value  in  simulation  was  309.00 


E9 


200.00  250.00  300.00  350.00  400.00 


Assumption:  F9 


Triangular  distribution  with  parameters: 
Minimum  2CX3.00 

Likeliest  330.CX) 

Maximum  400.00 

Selected  range  is  from  200.00  to  4(30.00 
Mean  value  in  simulation  was  31 1 .04 


F9 


200.00  250.00  300.00  350.00  400.00 


Assumption:  Compuserv  Account 


Normal  distribution  vwth  parameters: 

Mean  200.(X) 

Standard  Dev.  30.00 

Selected  range  is  from -Infinity  to  -rtnfinity 
Mean  value  in  simulation  was  201.44 


Computorv  Account 


110.00  155.00  200.00  245.00  290.00 


Assumption:  C10 


Norma!  distribution  with  parameters: 

Mean  2(30.(X) 

Standard  Dev.  30.(30 

Selected  range  is  from -Infinity  to  4nfinity 
Mean  value  in  simulation  was  199.19 


CIO 


110.00  155.00  200.00  245.00  290.00 


1 
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Assumption:  D10 


Normal  distribution  with  parameters: 

Mean  200.00 

Standard  Dev.  30.00 

Selected  range  isfrom-lnfinityto  4nfinit/ 
Mean  value  in  simulation  was  199.76 

Assumption:  E10 


DtO 


Normal  distribution  with  parameters: 

Mean  200.0 

Standard  Dev.  30.0 

Selected  range  isfrom-lnfinityto  4nfinity 
Mean  value  in  simulation  was  200.39 


E10 


110.00  155.00  200.00  245.00  290.00 


Assumption:  F10 


Normal  distribution  with  parameters: 

Mean  200.0 

Standard  Dev.  30.0 

Selected  range  is  from  -Infinit/to  -linfinit/ 
Mean  value  in  simulation  was  199.89 
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Assumption:  sales  yr  2 

Normal  distribution  with  parameters: 

Mean  1,CXX).00 

Standard  Dev.  250.00 

Selected  range  is  from -Infinity  to  -(infinity 
Mean  value  in  simulation  was  991.87 

Correlated  vwth: 
sales  yr  3  (D13) 
sales  yr  4  (E13) 


1,000.00  1.375.00  1,750.00 


Assumption:  sales  yr  3 


Normal  distribution  with  parameters: 

Mean  2,500.00 

Standard  Dev.  450.00 

Selected  range  is  from  -Infinity  to  -(infinity 

Mean  value  in  simulation  wras  2,496.19 

Correlated  with: 

sales  yr  2  {Cl 3) 
sales  yr  4  (E13) 
sales  yr  5  (F13) 


2.SOO.OO  3.175.00  3,860.00 
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Assumplion:  sales  yr  4 


Normal  distribution  wilh  parameters: 

Mean  3,500.00 

Standard  Dev.  550.00 

Selected  range  isfrom-lnfinit/to  -rfnfinily 

Mean  value  in  simulation  was  3,488.02 

Correlated  with: 
sales  yr  2  {C13) 
sales  yr  3  (D13) 
sales  yr  5  (FI 3) 


sales  yr  4 


0.13 

0.33 

0.50 


Assumption;  sales  yr  5 


Normal  disfribution  v«th  parameters: 

Mean  2,500.00 

Standard  Dev.  450.00 

Selected  range  is  from  -Infinity  to  -(Infinity 
Mean  value  in  simulation  was  2,504.80 

Correlated  with: 

sales  yr  3  (D13)  0.15 

salesyr4  (E13)  0.50 


sales  yr  5 
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CRDA  Trackinsf  Number : 
rnmpanv  Tnvolved  :  Paragon  Aircraft 
Point  of  Contact/Office  Svm/Phone  #  : 
Purpose: 


Identify  All  the  expected  benefits  received  from  the  CRD  A: 

1 .  Expanding  the  data  base  on  composites  (new  material  being  used  for  Paragon) 

*  One  man  year  of  effort  saved  in  the  development  of  data  base  and  knowledge 
transfer— savings  estimated  between  90  -  125  k  in  salary  that  would  have  been  paid  to  WL 
engineer 

2.  Knowledge  transfer  back  to  the  unit  that  would  have  taken  at  least  one  man  year  to 
obtain 

3.  Consultation  with  collaborator  during  FAA  certification  and  approval  process 

4.  Knowledge  gained  in  adhesively  bonded  structures 

5.  Royalty  payments  to  WL  Materials  Branch  on  the  first  ten  aircraft  developed  and  sold 
under  the  agreement-$10,120  paid  to  WL  on  each  aircraft  sold.  This  royalty  stream  will 
be  used  for  time  and  services  rendered  by  WL  engineers. 

Provide  a  quantified  estimate  of  each  benefit  mentioned  above  in  terms  of  dollars 
saved/received,  man  hr.  saved,  or  time  saved. 


CRDA# 

Company : 

Expected  Benefits 
Aircraft  sold  by  Paragon 
$10,120  per  aircraft  sold 
Man  Year  1  work  saved 
Man  Year  2  work  saved 
Total  Revenue 
Present  Value  at  7% 


PARAGON  Aircraft  Development  of  Light  Commuter  Plane 


Total  PV 


Year  1  Year  2 

0 
0 

-'lOS, 422 
$  86,583 

. $  '192,005  $ 

$  179,444  $ 


Years  Year  4  Year  5  Revenues 

0  1.66286268  4.76313687  2.33819591 
0  1682871704  4820279451  ”23662.5426 


-  $  16,828  $  48,203  $  23,663 

-  $  13,737  $  36,774  $  16,871  $246,826 


I 


A 


-  Man  year  1  represents  the  work  savings  resulting  form  Paragon’s  involvement  in  data 
base  expansion  on  composites,  knowledge  transfer  to  WL  in  seeing  composite  structures 
built,  and  consultation  with  collaborator  during  FAA  approval  process. 

-  Man  year  2  represents  the  work  savings  associated  with  advertising,  marketing,  and 
finding  an  outside  partner  willing  to  work  with  and  test  the  ISO  grid  composite  structure. 
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Identify  all  of  the  benefits  received  through  CRDA  relationship  that  were  not 
originally  spelled  out  in  the  agreement: 

1 .  WL  found  a  use  for  the  low  tech  ISO  grid  composite  design-a  developed  structure  can 
now  be  used  by  an  outside  partner  and  performance  data  can  be  obtained  at  no  cost  to 
WL. 

*  One  man  year  of  marketing  and  advertising  effort  saved  by  finding  a  customer 
for  the  ISO  grid  structure 

2.  Fabrication  to  take  place  at  the  Dayton  International  Airport-logistics  an  systems 
support  may  be  provided  by  WL  and  we  will  get  the  use  of  an  airplane  test  bed. 


Identify  any  nonquantifiable  benefits  received  by  the  Air  Force  as  a  result  of  the 
CRDA: 

-  Not  just  a  testing  agreement;  a  great  deal  of  knowledge  obtained  during  this  CRDA  that 
will  benefit  WL. 


Crystal  Ball  Report 

Simulation  started  on 
7/14/96  at  7:36:32 
Simulation  stopped  on 
7/14/96  at  7:36:55 


Forecast:  G1 1 
Summary 

Display  Range  is  from  $200,000  to  $300,000  Present  Value 
Dollars 

Entire  Range  is  from  $192,733  to  $293,790  Present  Value 
Dollars 

After  1,000  Trials,  the  Std.  Error  of  the  Mean  is  $545 


Statistics 

Value 

Trial 

s 

1000 

Mea 

n 

$246,826 

Median  (approx.) 

$248,718 

Mode  (approx.) 

$254,883 

Standard 

$17,223 

Deviation 

Variance 

$296,639,136 

Skewness 

-0.32 

Kurtosis 

2.73 

Coeff.  of 

0.07 
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Variability 
Range  Minimum 
Range  Maximum 
Range  Width 
Mean  Std.  Error 


$192,733 

$293,790 

$101,057 

$544.65 


Percentiles: 


Percentile 

0% 

10% 

20% 

30% 

40% 

50% 

60% 

70% 

80% 

90% 

100% 


Present  Value  Dollars 
(approx.) 

$192,733 

$222,825 

$231,808 

$238,239 

$243,219 

$248,718 

$253,367 

$257,129 

$261,721 

$266,937 

$293,790  * 


Assumption:  Aircraft  sold  in  year  3 


Triangular  distribution  with  parameters: 


Minimum 

0.00 

Likeliest 

2.00 

Maximum 

3.00 

Selected  range  is  from  O.OOto  3.00 
Mean  value  in  simulation  was  1.66 


no 


0.50 


Correlated  with: 

Aircraft  sold  in  year  4  (E6) 

Aircraft  sold  in  year  5  (F6) 

0.15 


t 


1 


Assumption:  Aircraft  sold  in  year  4 


Triangular  distribution  with  parameters: 


Minimum 

0.00 

Likeliest 

6.00 

Maximum 

8.00 

Selected  range  is  from  O.OOto  8.00 
Mean  value  in  simulation  was  4.76 


Aircraft  told  in  year  4 


0.00  2.00  4.00  e.OO  8.00 


Correlated  with: 

Arcraft  sold  in  year  3  (D6)  0.50 

Arcraftsold  in  year  5  (F6)  -0.33 


Assumption:  Aircraft  sold  in  year  5 


Triangular  disiribulion  with  parameters: 


Minimum 

0.00 

Likeliest 

2.00 

Ma)dmum 

5.00 

Selected  range  is  from  O.OOto  5.00 
Mean  value  in  simulation  was  2.34 


Aircrafft  sold  in  year  S 


0.00  1.25  2.50  3.75  5.00 


Correlated  with: 

Arcraftsold  in  year  3  (D6)  -0.15 

Arcraftsold  in  year  4  (E6)  -0.33 


Assumption:  Man  Year  1  work  saved 


Triangular  disfribution  with  parameters: 
Minimum  $90,000 

Likeliest  $100,000 

Maximum  $125,000 


Selected  range  is  from  $90,000 to  $125,000 
Mean  value  in  simulation  was  $105,422 
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Assumption:  Man  Year  2  work  saved 

Triangular  distribution  with  parameters: 
Minimum  $75,000 

Likeliest  $85,000 

Maximum  $100,000 

Selected  range  is  from  $75,000to  $100,000 
Mean  value  in  simulation  was  $86,583 


n 


t 


Man  Year  2  work  saved 


$75,000  $81,250  $87,500  $93,750  $100,000 
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CRDA  Tracking  Number  :  93  -  335 -WL  -  01 


Company  Involved  ;  Wright  State  University 
Point  of  Contact/Office  Sym/Phone  #  :  Ed  Stuts  /  5-8658 

Purpose:  Use  government  facilities  and  expertise  to  help  a  university  carry  out  research 
and  testing  requests  by  industrialized  organizations  and  other  governmental  agencies 

Identify  All  the  expected  benefits  received  from  the  CRDA: 

1 .  WSU  paid  WL  for  test  facilities  and  expertise  provided  by  WL  and  WL  personnel 
--  WSU  requested  information,  expertise,  or  resources  from  WL-WL  was  not  paid  for 
each  research  request  but  was  paid  in  lump  sums  for  research  that  was  within  the  scope 
of  the  WL  mission. 

—  WL  effort  towards  the  CRDA  was  limited  to  only  research  that  was  mutually 
beneficial  to  WL  and  WSU 


2.  Advancement  of  research  and  access  to  data  in  technical  areas 

Provide  a  quantified  estimate  of  each  benefit  mentioned  above  in  terms  of  dollars 
saved/received,  man  hr.  saved,  or  time  saved. 


Estimated  cash  flows  for  years  97  and  98  discounted  using  7%  interest  rate. 

CRDA  #  93  -  335  -WL  -  01 


Company:  Wright  State  University 

Year :  Nov-94  Apr-95 

Revenues :  $  4,609  $  17,120 
Disc.  CS 


Total 

May-96  Revenues 

$  22,365  $  19,948  $  20,045  $80,245 

$18,643.05  $17,508.26 


Identify  all  of  the  benefits  received  through  CRDA  relationship  that  were  not 
originally  spelled  out  in  the  agreement: 


-  Money  received  through  CRDA  used  for  capital  improvements  at  WL— mission  at  WL 
has  been  helped  by  these  expenditures  because  the  equipment  that  was  purchased  (special 
lasers)  wouldn’t  have  been  purchased  due  to  funding  limitations.-- WL  has  been  able  to 
due  research  that  wouldn’t  have  been  possible  without  equipment  expenditures. 

Identify  any  nonquantifiable  benefits  received  by  the  Air  Force  as  a  result  of  the 
CRDA: 


-  The  fact  that  WL  has  been  effectively  transferring  technology  to  an  outside  partner. 

Identify  any  lessons  learned  about  tbe  CRDA  process  during  the  development  of  this 
CRDA 

1 .  WL  already  had  an  ongoing  contract  with  WSA-the  bond  between  the  two  agencies 
made  them  more  willing  to  adhere  to  the  agreement  and  make  it  work. 
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2.  The  objectives  for  this  CRD  A  were  very  open  and  general— this  gave  WL  the  freedom 
to  pursue  a  wide  range  of  activities  under  this  CRDA-WL’s  main  requirement  was  that 
the  activity  bring  in  revenue  and  fall  under  the  scope  of  the  WL  mission. 

3 .  WL  had  the  freedom  to  pick  and  chose  the  research  they  were  willing  to  perform 
under  this  CRD  A. 


Crystal  Ball  Report 
Simulation  started  on  7/13/96  at 
14:34:30 

Simulation  stopped  on  7/13/96  at 
14:34:42 

Forecast:  Revenues  Cell: 

G5 

Summary 

Display  Range  is  from  $72,500  to  $97,500  Dollars 
Entire  Range  is  from  $68,998  to  $98,624 
Dollars 

After  1,000  Trials,  the  Std.  Error  of  the  Mean  is  $132 


Statistics 

Value 

Trial 

1000 

s 

Mea 

$84,087 

n 

Median  (approx.) 

$84,233 

Mode  (approx.) 

$84,848 

Standard 

$4,185 

Deviation 

Variance 

$17,511,090 

Skewness 

-0.02 

Kurtosis 

3.05 

Coeff.  of 

0.05 

Variability 

Range  Minimum 

$68,998 

Range  Maximum 

$98,624 

Range  Width 

$29,626 

Mean  Std.  Error 

$132.33 
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Cell  G5 


Forecast:  Revenues 
Frequency  Chart 


998  Trials  Shown 


Forecast:  Revenues  (cont'd) 


Percentiles: 


Percentile 

0% 

10% 

20% 

30% 

40% 

50% 

60% 

70% 

80% 

90% 

100% 


Dollars  (approx.! 
$68,998 
$78,893 
$80,717 
$81,816 
$83,004 
$84,233 
$85,122 
$86,304 
$87,728 
$89,390 
$98,624 


Normal  distribution  with  parameters: 

Mean  $20,000 

Standard  Dev.  $3,000 

Selected  range  is  from -Infinity  to  -Hnfinity 
Mean  value  in  simulation  was  $19,948 


Normal  distribution  with  parameters: 

Mean  $20,000 

Standard  Dev.  $3,000 

Selected  range  is  from  -Infinity  to  -infinity 
Mean  value  in  simulation  was  $20,045 


Frequency 
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